Automatic proxy selection in concurrent requests

[harvetech]

Dear all,

I am looking for a way by which I can configure crawlee to make multiple concurrent requests to scrap a website but each request should use a different proxy or there should be some intelligent ways that crawlee could select proxies from the available list of proxies. The reason I want to do this is to avoid overloading a proxy and eventually getting banned.

I want to scrap the same website's 100 pages concurrently. Now If all 100 requests are going from a single proxy it will for sure get banned. How does crawlee deals with this issue ?

I am sorry am unable to find clear answers in the documentation.

[B4nan]

This works based on sessions, e.g. you can set up the crawler to only use proxy once by retiring the session after the first usage.

https://crawlee.dev/docs/guides/session-management

With browsers, proxy is tied to the browser context, which basically means a browser window with all its tabs. You can set crawlee to only use one tab per browser instance, or use proxyPerPage option - note that by doing this, caching of resources won't work, as each page will be processed via anonymous browser window, and those don't have any resource caching between each request. We also have an experimental containers feature that should help to get around it, but it is quite limited, AFAIK it works only with firefox.

Lastly, you can provide custom logic in the proxy configuration options:

https://crawlee.dev/api/core/interface/ProxyConfigurationOptions