Skip to content

1.1.0

Latest
Latest
Compare
Choose a tag to compare
@github-actions github-actions released this 13 Aug 14:20
· 2 commits to main since this release
1.1.0
This tag was signed with the committer’s verified signature.
freddyheppell Freddy Heppell
SSH Key Fingerprint: g2e9HLT9VrdorujIdE24Ve3vFoQNWtyzUWdcpzpZIzo Learn about vigilant mode.
8328435
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Learn about vigilant mode.

Features & Improvements

  • WPextract is now completely type-annotated, and ships a py.typed file to indicate this
  • Added --user-agent argument to wpextract download to allow customisation of the user agent string
  • HTTP errors raised when downloading now all inherit from a common HTTPError class
  • If an HTTP error is encountered while downloading, it will no longer end the whole scrape process. A warning will be logged and the scrape will continue, and if some data was obtained for that type, it will be saved as normal. HTTP transit errors (e.g. connection timeouts) will still end the scrape process.
  • Improved the resiliency of HTML parsing and extraction by better checking for edge cases like missing attributes
    • Translation picker extractors will now raise an exception if elements are missing during the extraction process.
  • Simplified the WordPress API library by removing now-unused cache functionality. This will likely improve memory usage of the download process.
  • Significantly more tests have been added, particularly for the download process

Fixes

  • Fixed the scrape crawling step crashing if a page didn't have a canonical link or og:url meta tag
  • Fixed the scrape crawling not correctly recognising when duplicate URLs were encountered. Previously duplicates would be included, but only one would be used. Now, they will be correctly logged. As a result of this change, the SCRAPE_CRAWL_VERSION has been incremented, meaning running extraction on a scrape will cause it to be re-crawled.
  • Fixed the return type annotation LangPicker.get_root(): it is now annotated as (bs4.Tag or None) instead of bs4.PageElement. This shouldn't be a breaking change, as the expected return type of this function was always a Tag object or None.
  • Type of TranslationLink.lang changed to reflect that it can accept a string to resolve or an already resolved Language instance
  • Fixed downloading throwing an error stating the WordPress v2 API was not supported in other error cases
  • Fixed the maximum redirects permitted not being set properly, meaning the effective value was always 30

Documentation

  • Improved guide on translation parsing, correcting some errors and adding information on parse robustness and performance
Assets 6
Loading