PuppeteerCrawler: handleRequestFunction failed, reclaiming failed request back to the list or queue

[panxiaomeng]

hi，
when I run the command npm run crawler in the container, it prints below error, could you please help check what happened?

Writing outputs to file './outputs/output.txt'.
INFO System info {"apifyVersion":"1.0.2","apifyClientVersion":"1.0.5","osType":"Linux","nodeVersion":"v12.22.12"}
WARN Neither APIFY_LOCAL_STORAGE_DIR nor APIFY_TOKEN environment variable is set, defaulting to APIFY_LOCAL_STORAGE_DIR="/mountdir/a4a055d1-30fe-470f-b2f6-df4c47304e80/crawler/dock
er-services/docsCrawler/apify_storage"
Starting Crawler.
INFO PuppeteerCrawler:AutoscaledPool:Snapshotter: Setting max memory of this run to 8028 MB. Use the APIFY_MEMORY_MBYTES environment variable to override it.
INFO PuppeteerCrawler:AutoscaledPool: state {"currentConcurrency":0,"desiredConcurrency":2,"systemStatus":{"isSystemIdle":true,"memInfo":{"isOverloaded":false,"limitRatio":0.2,"act
ualRatio":null},"eventLoopInfo":{"isOverloaded":false,"limitRatio":0.6,"actualRatio":null},"cpuInfo":{"isOverloaded":false,"limitRatio":0.4,"actualRatio":null},"clientInfo":{"isOver
loaded":false,"limitRatio":0.3,"actualRatio":null}}}
ERROR PuppeteerCrawler: handleRequestFunction failed, reclaiming failed request back to the list or queue {"url":"https://developer-test.internal.meng.com","retryCount":1,"id":"
MNQgqyvflbK5tY4"}
Error: browserController.newPage() failed: HJouQ9DU1l7DboRwV9c-t
Cause:browserController.newPage() timed out..
at BrowserPool._createPageForBrowser (/mountdir/a4a055d1-30fe-470f-b2f6-df4c47304e80/crawler/docker-services/docsCrawler/node_modules/browser-pool/src/browser-pool.js:384:19)
ERROR PuppeteerCrawler: handleRequestFunction failed, reclaiming failed request back to the list or queue {"url":"https://developer-test.internal.meng.com","retryCount":2,"id":"
MNQgqyvflbK5tY4"}
Error: browserController.newPage() failed: XvHSGQC8IeORglwsu_9jW
Cause:browserController.newPage() timed out..

[B4nan]

Cause:browserController.newPage() timed out..

Your problem is not reclaiming of the request, but newPage timing out.

Can you provide some reproduction?

[pocesar]

this is an old issue with SDK versions 1.x. upgrade to at least 2 and it should be good

[B4nan]

Ouch totally ignored the sdk version, good catch!

[panxiaomeng]

@pocesar , I am not familiar with this, could you please let me know how to configure the SDK?

by the way, I just did below action, and it looks solved this issue.

wget -q -O - https://dl-ssl.google.com/linux/linux_signing_key.pub | apt-key add - && echo "deb http://dl.google.com/linux/chrome/deb/ stable main" >> /etc/apt/sources.list.d/google.list
apt-get update
apt-get install -yq gconf-service libasound2 libatk1.0-0 libc6 libcairo2 libcups2 libdbus-1-3 libexpat1 libfontconfig1 libgcc1 libgconf-2-4 libgdk-pixbuf2.0-0 libglib2.0-0 libgtk-3-0 libnspr4 libpango-1.0-0 libpangocairo-1.0-0 libstdc++6 libx11-6 libx11-xcb1 libxcb1 libxcomposite1 libxcursor1 libxdamage1 libxext6 libxfixes3 libxi6 libxrandr2 libxrender1 libxss1 libxtst6 ca-certificates fonts-liberation libappindicator1 libnss3 lsb-release xdg-utils wget x11vnc x11-xkb-utils xfonts-100dpi xfonts-75dpi xfonts-scalable xfonts-cyrillic x11-apps

after these actions, no issue I stated in the description,

however there are a lot of web pages that can't be crawled (which can be crawled in the server, not the container)
the error is like this:

Crawl status: 51 url(s) crawled in 19 minutes.
ERROR PuppeteerCrawler: handleRequestFunction failed, reclaiming failed request back to the list or queue {"url":"https://developer-test.internal.meng.com/docs/edf/contributing/code-contributions/working-with-feature-branches/","retryCount":3,"id":"1RfMi83whKSmdfT"}
Error: gotoFunction timed out after 10 seconds.
at Object.handleRequestTimeout (/mountdir/a5fcf388-cfc4-4179-85c9-9f25a919a8eb/crawler/docker-services/docsCrawler/node_modules/apify/build/crawlers/crawler_utils.js:15:11)
at PuppeteerCrawler._handleNavigationTimeout (/mountdir/a5fcf388-cfc4-4179-85c9-9f25a919a8eb/crawler/docker-services/docsCrawler/node_modules/apify/build/crawlers/browser_crawler.js:378:29)
at PuppeteerCrawler._handleNavigation (/mountdir/a5fcf388-cfc4-4179-85c9-9f25a919a8eb/crawler/docker-services/docsCrawler/node_modules/apify/build/crawlers/browser_crawler.js:362:18)

successful crawling:
Crawling url 'https://developer-test.internal.meng.com/docs/edf/contributing/code-contributions/working-with-feature-branches/'.
Data miner is processing data for url 'https://developer-test.internal.meng.com/docs/edf/contributing/code-contributions/working-with-feature-branches/'.
Persisting data for url 'https://developer-test.internal.meng.com/docs/edf/contributing/code-contributions/working-with-feature-branches/' with title 'Working with feature branches - Contributing to EDF | EDF'.
Writing content for url 'https://developer-test.internal.meng.com/docs/edf/contributing/code-contributions/working-with-feature-branches/' to file.
Finished processing url 'https://developer-test.internal.meng.com/docs/edf/contributing/code-contributions/working-with-feature-branches/'.

could you give some suggestions? is it because this pages can't be accessed?

[pocesar]

you can try increasing the timeout, or debug why it's timing out. if you have waitUntil set to load or networkidle0, the timeout might be too short for your use case

[lloydjatkinson]

This is still a problem today with the latest version

[B4nan]

Could you provide complete reproduction for this?

[germanattanasio]

I'm seeing this when trying to crawl http://www.docs.sonarcloud.io/digging-deeper/activity-and-history/ which returns a DNS error

[B4nan]

That URL is apparently wrong, the domain does not support www. prefix. This URL seems to be working fine:

https://docs.sonarcloud.io/digging-deeper/activity-and-history/

[germanattanasio]

Interesting. Thanks @B4nan

[aditodkar]

@B4nan I am getting similar error: https://console.apify.com/view/runs/vzn1cFAcdAJLJMANY All domains are working fine not sure why apify is not executing it.

[B4nan]

That actor is using really outdated SDK version (0.20.2), from the times before crawlee actually existed. Try asking in the issues tab of that actor.

[aditodkar]

Ok got it. I think author is not active. I will fork and create updated version of it. Thanks for the info.