Skip to content

New name, improved scheduling and other

Compare
Choose a tag to compare
@sibiryakov sibiryakov released this 15 Apr 13:19
· 778 commits to master since this release
  • Frontera is the new name for Crawl Frontier.
  • Signature of get_next_requests method is changed, now it accepts arbitrary key-value arguments.
  • Overused buffer (subject to remove in the future in favor of downloader internal queue).
  • Backend internals became more customizable.
  • Scheduler now requests for new requests when there is free space in Scrapy downloader queue, instead of waiting for absolute emptiness.
  • Several Frontera middlewares are disabled by default.