PlaywrightCrawler - Possible to pause the crawl in case of 403 blocking

[yossi-s]

Hey, I have a question related to the PlaywrightCrawler.
Is it possible to pause the crawl in case I started to get 403 due to rate limit violation? wait for 2 min and back to crawl again?

Also the url's that was not succeed to crawled due to this blocking will back to the queue?
Thanks!

[harshmaur]

You can first define sessionPoolOptions in your crawler like this

then you can handle the 403 as a middleware in your routes.

[yossi-s]

Not sure I got it.
Do you have an example for that?

[yossi-s]

Is it possible if you can paste here an example how to create a middleware that catch the blocked requests, pause the crawler for some amount of time (lets say 2 min) and resume to work after that.

[RobinHoutevelts]

Also unclear to me how to handle rate limits etc.
I don't have a pool of IPs available

[B4nan]

The best way to handle them is not to trigger them, try limiting the max concurrency so you do not bombard the target site too much.

[RobinHoutevelts]

Thanks for your response! 🙏
I should have clarified why I want a rate limit instead of a concurrency limit:

I have a project that sits behind a CDN that supports stale-while-revalidate.
In short it means the CDN will respond within a few ms while asynchronously fetching an updated version in the background.

Even with a concurrency limit of 1 I'm causing issues because the crawler can move through the urls really fast due to the CDN, but the CDN is then hammering my origin when it fetches fresh data afterwards. Ideally I could instrument the CDN to ratelimit, but I can't.

I'm migrating from linkchecker which has a maxrequestspersecond=NUMBER option which I used to circumvent this.

Limit the maximum number of HTTP requests per second to one host. The average number of requests per second is approximately one third of the maximum. Values less than 1 and at least 0.001 can be used. To use values greater than 10, the HTTP server must return a LinkChecker response header. The default is 10. Command line option: none

[B4nan]

You can do the same here too, we have maxRequestsPerMinute for that.

https://crawlee.dev/docs/guides/scaling-crawlers#maxrequestsperminute

[RobinHoutevelts]

Aha! Thanks!!