Can't find document/discussion/issue about how to resume the crawling after ctrl-c without erasing the pulled dataset

[lubobill1990]

Since crawling is time costing, and it's common to stop the crawler intentionally or accidentally.

However, by default, the storage is removed when starting a new job.

Even by setting purgeOnStart to false, the files in storage/datasets/default are removed.

I believe pause/resume is a common feature of crawlers.

So, could anyone help on this question?

[B4nan]

Even by setting purgeOnStart to false, the files in storage/datasets/default are removed.

If it does not work for you, are you probably setting it too late (as its the very first async call that is touched some storage which triggers the clean up). Try using the CRAWLEE_PURGE_ON_START env var instead - which is exactly what the log message on pausing suggests. If that won't help, we need to see more details.

[DavidGOrtega]

@B4nan I have a similar question, If by any meaning my crawler stops I would like to not repeat already crawled urls. I have read the docs trying to find something regarding this issue but I had no luck.

To clarify: I want to not crawl or request urls that I have already in my Dataset.

[B4nan]

That's how it works with CRAWLEE_PURGE_ON_START disabled, if you don't purge the storage, it will continue where it stopped.

[DavidGOrtega]

CRAWLEE_PURGE_ON_START=false

effectively does it, however I would like to not continue but also restart from the main page and not process already gathered URLs.
This is quite useful in pages that generates fresh content where the entry point is always the same i.e. a forum

The only idea to handle this right now is to add a ts query param in the index page.

await crawler.run([`https://myforumexample.com?ts=${Date.now()}`]);