Replies: 1 comment
-
One thing you can always do, regardless of the crawler state, is to enqueue new links with a random |
Beta Was this translation helpful? Give feedback.
0 replies
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
-
while using the CRAWLEE_PURGE_ON_START=0, is there a way to re-crawl some of the pages?
for examlple,
crawling on a list page, and looking for new products.
so you dont want to crawl the product again, but you want to crawl on the list pages to find new products when this list is changing.
i tries using skipNavigation: false without success.
Beta Was this translation helpful? Give feedback.
All reactions