Execution context was destroyed, most likely because of a navigation

[teammakdi]

Which package is this bug report for? If unsure which one to select, leave blank

@crawlee/puppeteer (PuppeteerCrawler)

Issue description

INFO  PuppeteerCrawler: Initializing the crawler.
Error handler:  Error: Execution context was destroyed, most likely because of a navigation.
    at rewriteError (/node_modules/puppeteer-core/lib/cjs/puppeteer/common/ExecutionContext.js:313:15)
    at runNextTicks (node:internal/process/task_queues:60:5)
    at process.processImmediate (node:internal/timers:447:9)
    at process.callbackTrampoline (node:internal/async_hooks:130:17)
    at async #evaluate (/node_modules/puppeteer-core/lib/cjs/puppeteer/common/ExecutionContext.js:252:60)
    at async IsolatedWorld.document (/node_modules/puppeteer-core/lib/cjs/puppeteer/common/IsolatedWorld.js:112:26)
    at async IsolatedWorld.$ (/node_modules/puppeteer-core/lib/cjs/puppeteer/common/IsolatedWorld.js:100:26)
    at async Promise.all (index 0)
    at async PuppeteerCrawler.isRequestBlocked (/node_modules/@crawlee/browser/internals/browser-crawler.js:201:17)
    at async PuppeteerCrawler._runRequestHandler (/node_modules/@crawlee/browser/internals/browser-crawler.js:249:17)
WARN  PuppeteerCrawler: Reclaiming failed request back to the list or queue. Execution context was destroyed, most likely because of a navigation.
 {"id":"8HUD9HwtCc0t4FM","url":"https://www.quizerry.com/category/computer-science/","retryCount":1}
Error handler:  Error: Execution context was destroyed, most likely because of a navigation.
    at rewriteError (/node_modules/puppeteer-core/lib/cjs/puppeteer/common/ExecutionContext.js:313:15)
    at runNextTicks (node:internal/process/task_queues:60:5)
    at process.processImmediate (node:internal/timers:447:9)
    at process.callbackTrampoline (node:internal/async_hooks:130:17)
    at async #evaluate (/node_modules/puppeteer-core/lib/cjs/puppeteer/common/ExecutionContext.js:252:60)
    at async IsolatedWorld.document (/node_modules/puppeteer-core/lib/cjs/puppeteer/common/IsolatedWorld.js:112:26)
    at async IsolatedWorld.$ (/node_modules/puppeteer-core/lib/cjs/puppeteer/common/IsolatedWorld.js:100:26)
    at async Promise.all (index 0)
    at async PuppeteerCrawler.isRequestBlocked (/node_modules/@crawlee/browser/internals/browser-crawler.js:201:17)
    at async PuppeteerCrawler._runRequestHandler (/node_modules/@crawlee/browser/internals/browser-crawler.js:249:17)
WARN  PuppeteerCrawler: Reclaiming failed request back to the list or queue. Execution context was destroyed, most likely because of a navigation.
 {"id":"8HUD9HwtCc0t4FM","url":"https://www.quizerry.com/category/computer-science/","retryCount":2}

Code sample

const { PuppeteerCrawler, Configuration } = require('crawlee')

const crawleeConfig = Configuration.getGlobalConfig()
crawleeConfig.set('persistStorage', false)

function getPageViewPort (page) {
  if (!page || page.isClosed()) return null
  const { width, height } = page.viewport()
  return `${width}x${height}`
}

async function getPageUserAgent (page) {
  if (!page || page.isClosed()) return null
  return await page.evaluate(() => navigator.userAgent)
}

const crawler = new PuppeteerCrawler({
  requestHandlerTimeoutSecs: 120,
  retryOnBlocked: true,
  keepAlive: true,
  sameDomainDelaySecs: 5,
  launchContext: {
    useChrome: true,
    useIncognitoPages: true,
    launchOptions: {
      headless: true,
      ignoreHTTPSErrors: true
    }
  },
  autoscaledPoolOptions: {
    minConcurrency: 1,
    desiredConcurrency: 1,
    maxConcurrency: 1
  },
  requestHandler: async ({ request, page, response }) => {

    const pageUserAgent = await getPageUserAgent(page)
    const pageViewPort = getPageViewPort(page)

    console.log(pageUserAgent, pageViewPort)

    console.info(`Successfully crawled ${response.url()} with code ${response.status()}`)
  },
  errorHandler: ({ request, response }, err) => {
    console.error('Error handler: ', err)
  },
  failedRequestHandler: async ({ request, page }, err) => {
    try {
      console.error('Failed request handler: ', err)
    } catch (error) {
      console.error('Error in failed request handler: ', error)
    }
  },
  preNavigationHooks: [
    async (crawlingContext, gotoOptions) => {
      gotoOptions.waitUntil = 'domcontentloaded'
    }
  ]
})

const url = 'https://www.quizerry.com/category/computer-science/'
crawler.run([
  {
    url,
    userData: {
      url
    }
  }
])

Package version

3.5.4

Node.js version

18.17.0

Operating system

Mac OS

Apify platform

• Tick me if you encountered this issue on the Apify platform

I have tested this on the next release

No response

Other context

The code is just a small POC.

[teammakdi]

Works fine if retryOnBlocked is set to false

[B4nan]

Sounds like a clash between retryOnBlocked and useIncognitoPages, please take a look @barjin

[barjin]

Hello @teammakdi, it's actually because of the gotoOptions.waitUntil = 'domcontentloaded' line. By default, our crawlers wait until all the resources on the page are loaded (load event), so these exact errors don't happen. You crawler works just fine if you remove this line.

In case you want to save some bandwidth and time and not wait to load image, CSS, and font files, you can use the blockRequests context helper in the preNavigation hook like this:

preNavigationHooks: [
   async ({ blockRequests }) => await blockRequests(),
],

You can read more about this helper in our documentation.

Please let us know if this helped. Thank you!