Enqueued link gets ignored

[nineBananas]

Could you, please, help me figure out why only one url gets processed and the second one gets ignored? I want to process the second link too. Here is the code:

import { CheerioCrawler } from 'crawlee';

const crawler = new CheerioCrawler({
	async requestHandler({ request, enqueueLinks, log }) {
		log.info(request.loadedUrl!);
		
		await enqueueLinks({
			urls: ['https://www.bricklink.com/ajax/clone/catalogifs.ajax?itemid=123906&iconly=0&pi=2']
		});
	}
});

await crawler.run([{
	url: 'https://www.bricklink.com/ajax/clone/catalogifs.ajax?itemid=123906&iconly=0'
}]);

The only difference between the two urls is an additional field pi at the end.

Platform: Mac OS 14.2.1, [email protected], node v21.5.0.

[B4nan]

They have the same uniqueKey so they get deduplicated (it ignores the query parameters), you can override that when enqueuing.

[nineBananas]

Thanks! Unfortunately, it is still not working, even though I tried two things and neither of them worked. First, I set uniqueKey to "first" for the first link. Second, I set uniqueKey of the second link to "second" with transformRequestFunction. Here is the full code:

const crawler = new CheerioCrawler({
	async requestHandler({ request, enqueueLinks, log }) {
		log.info(request.loadedUrl!);
		
		await enqueueLinks({
			urls: ['https://www.bricklink.com/ajax/clone/catalogifs.ajax?itemid=123906&iconly=0&pi=2'],
			transformRequestFunction: (request) => {
				request.uniqueKey = 'second';
				return request;
			}
		});
	}
});

await crawler.run([{
	url: 'https://www.bricklink.com/ajax/clone/catalogifs.ajax?itemid=123906&iconly=0',
	uniqueKey: 'first'
}]);

What did I do wrong?

[B4nan]

enqueueLinks is a helper for finding URLs on the current page, if you want to add a URL to the queue that you already have somewhere, I'd just do crawler.addRequests(). Could be related, as what you are doing with enqueueLinks is not really tested anywhere (as it's not the suggested usage).

[nineBananas]

OK, thanks. One last question. If a router is defined in a file separate from a crawler, what is the best way to access the request queue from a router handler? At the moment, I export the crawler with export, import it to the file with the router and use this code to add a request with all the parameters:

await crawler.addRequests([{
	url: url,
	label: 'NEW',
	userData: {
		first: 'string',
		second: 13
	}
}]);

[B4nan]

You have the crawler instance in the crawling context, you can access the RQ from it if needed (or just call crawler.addRequests on it).

https://crawlee.dev/api/core/interface/CrawlingContext#crawler

[nineBananas]

Thank you very much!

[ShaleenAg]

Hey @B4nan , not sure if its a bug but even arbitrary urls such as a https://google.com are not enqueued, we have shifted to using crawler.addRequests but just wanted to point this out in this context

[B4nan]

Not sure what you mean, but note that by default, enqueueLinks only adds links that point to the same domain (as the request you are enqueuing from). And this is currently also checked after redirects (so even if you add a URL that is on the same domain and redirects outside of it, such request will be ignored).

You shouldn't be using enqueue links for that, its a helper to enqueue links from the current page, but if you really want to, try switching the strategy to EnqueueStrategy.All.

https://crawlee.dev/api/core/enum/EnqueueStrategy

[ShaleenAg]

ah, makes sense, we'll stick with addRequests, I was just curious why it wasn't working, since we were just calling the API in our first request and then enqueuing actual links from it.