How to ignore certain error code?

[mbledkowski]

Hello,
I am trying to scrape cloudflare protected website.
I get 403 error everytime I make a connection, but after that I get connected in browser.
But the scraper is killing page right after it gets this error - Reclaiming failed request back to the list or queue. Request blocked - received 403 status code..
How can I add 403 status code to ignored ones?
Best regards.
Maciej Błędkowski

[LeMoussel]

Adding sessionPoolOptions: { blockedStatusCodes: [] }, to crawler options may solve your problem.

const crawler = new PuppeteerCrawler({
  sessionPoolOptions: { blockedStatusCodes: [403] },
  // ...
}); 

[mbledkowski]

I am sorry, but it does not solve my issue :/

[AndreyBykov]

Actually, the message and the snippet contradict each other and it might be confusing.

@mbledkowski I assume you added the code from the snippet with explicit 403 code mentioned there? While what you need is to pass an empty array, as it's written in the message itself:

const crawler = new PuppeteerCrawler({
  sessionPoolOptions: { blockedStatusCodes: [] },
  // ...
}); 

[mbledkowski]

Actually, the message and the snippet contradict each other and it might be confusing.

@mbledkowski I assume you added the code from the snippet with explicit 403 code mentioned there? While what you need is to pass an empty array, as it's written in the message itself:

const crawler = new PuppeteerCrawler({
  sessionPoolOptions: { blockedStatusCodes: [] },
  // ...
}); 

Thank you very much, passing empty array seems to work for me. Can you explain to me how exactly does it work? Does passing an empty array means that no error codes would be blocked? If so is there any simple way to unlock only 403 error code? Thank you in advanced.

[AndreyBykov]

Yes, this is currently a workaround - cloudflare started to return code 403 even for the pages, that are not blocked, but the ones that are being checked (like in your case). This should be addressed in future releases, but currently - this is the easiest option.