Crawl with puppeteer or cheerio based on url

[abhisheksurve45]

Hi

I have a use case to run crawlee in both browser and curl crawling mode, i.e. puppeteer and got.

So each url will hold crawling mode property, and based on mode, will do either puppeteer or got crawl.

Just wanted to know is this possible in crawlee? If yes, is it possible to achieve it using single crawlee instance / single request queue.

Thanks.

[B4nan]

You can use the puppeteer crawler, it has sendRequest helper in the context you can use to fire requests directly instead of a browser.

https://crawlee.dev/docs/examples/skip-navigation
https://crawlee.dev/api/puppeteer-crawler/interface/PuppeteerCrawlingContext#sendRequest
https://crawlee.dev/docs/guides/got-scraping#sendrequest-api

[abhisheksurve45]

I was looking for a solution which supports both puppeteer and cheerio based crawling.
@B4nan

[B4nan]

You can do that with the proposed sendRequest helper in combination with (yet another contextual helper) parseWithCheerio.

[abhisheksurve45]

@B4nan Tried what you had mentioned

const { PuppeteerCrawler, sleep } = require('crawlee')

const puppeteerExtra = require('puppeteer-extra')

const crawler = new PuppeteerCrawler({
  maxRequestRetries: 2,
  launchContext: {
    launcher: puppeteerExtra,
    launchOptions: {
      useIncognitoPages: true,
      headless: false
    }
  },

  maxRequestsPerCrawl: 5,
  async requestHandler ({ request, page, sendRequest, response }) {
    try {
      await sleep(3000)
      console.log('in requestHandler')
      if (request.skipNavigation) {
        await sendRequest()
      } else {
        console.log('already crawled')
      }
    } catch (err) {
      console.error('in catch requestHandler')
    }
  },
  errorHandler ({ request }, err) {
    console.log('in errorHandler')
  },
  failedRequestHandler ({ request }, err) {
    try {
      console.log('in failedRequestHandler')
      console.log(`Request ${request.url} failed ${request.retryCount} many times.`)
      console.error(err)
    } catch (err) {
      console.error('in failedRequestHandler catch')
    }
  },
  preNavigationHooks: [
    async (crawlingContext, gotoOptions) => {
      console.log('in preNavigationHooks')
    }
  ]
})

crawler.addRequests([{
  url: 'https://google.com',
  skipNavigation: true
},
{
  url: 'https://about.google/',
  skipNavigation: false
}])

crawler.run()

However my usecase is that I dont want to open browser/page for urls that are meant to be crawled via got scraping.

In this case it opens up a empty chrome tab for https://google.com url

[B4nan]

Not sure if that is your actual code - if it is, you are missing quite a lot of awaits in there.

skipNavigation is what drives this, requests with that enabled won't open the browser. Such requests are going directly to the request handler.

Not sure why are you adding the puppeteer-extra to the mix, it shouldn't be needed (and can generally have some unwanted side effects).

[abhisheksurve45]

@B4nan please ignore the awaits on crawler.addRequests and crawler.run. This is just an example code.

skipNavigation is what drives this, requests with that enabled won't open the browser. Such requests are going directly to the request handler.

This doesn't seem to be happening. If I keep my headless mode as false and set maxOpenPagesPerBrowser to 1
, I can see that chrome is opened twice. (chrome opened === concurrency)

[abhisheksurve45]

Script to test the above scenario

const { PuppeteerCrawler, sleep } = require('crawlee')

const crawler = new PuppeteerCrawler({
  maxRequestRetries: 1,
  browserPoolOptions: {
    maxOpenPagesPerBrowser: 1
  },
  launchContext: {
    launchOptions: {
      useIncognitoPages: false,
      headless: false
    }
  },
  minConcurrency: 2,
  maxConcurrency: 2,
  maxRequestsPerCrawl: 5,
  async requestHandler ({ request, page, sendRequest, response }) {
    try {
      await sleep(3000)
      console.log('in requestHandler')
      if (request.skipNavigation) {
        await sendRequest()
      } else {
        console.log('already crawled')
      }
    } catch (err) {
      console.error('in catch requestHandler')
    }
  },
  errorHandler ({ request }, err) {
    console.log('in errorHandler')
  },
  failedRequestHandler ({ request }, err) {
    try {
      console.log('in failedRequestHandler')
      console.log(`Request ${request.url} failed ${request.retryCount} many times.`)
      console.error(err)
    } catch (err) {
      console.error('in failedRequestHandler catch')
    }
  },
  preNavigationHooks: [
    async (crawlingContext, gotoOptions) => {
      console.log('in preNavigationHooks')
    }
  ]
})

crawler.addRequests([{
  url: 'https://google.com',
  skipNavigation: true
},
{
  url: 'https://about.google/',
  skipNavigation: false
}])

crawler.run()

[abhisheksurve45]

@B4nan let me know if any thoughts on this. Not urgent.

[abhisheksurve45]

@B4nan @barjin