How does crawlee work with node-schedule?

[faner11]

I am a newbie in crawlee. I need to monitor some fixed links. Often, node-schedule cannot continue because of an error in a task. The following is my code,Should I do something?

import { Configuration, PlaywrightCrawler, ProxyConfiguration } from 'crawlee'
import { scheduleJob } from 'node-schedule'

import { router } from './routes'

const createCrawler = () => {
  const config = new Configuration()
  const crawler = new PlaywrightCrawler(
    {
      proxyConfiguration: new ProxyConfiguration({ proxyUrls: [p] }),
      requestHandler: router,
      maxRequestsPerCrawl: 50,
      maxRequestRetries: 0,
    },
    config,
  )
  return crawler
}

const taskRun = async () => {
  const crawler = createCrawler()
  const linkList = ['https://github.com/xx1', 'https://github.com/xx2']
  await crawler.run(linkList).finally(() => {
    void purgeDefaultStorages()
  })
}

scheduleJob('* 0-4 * * *', async () => {
  await taskRun()
})

scheduleJob('*/2 5-10 * * *', async () => {
  await taskRun()
})

Also, can you tell me what is the difference between createCrawler for each task and using a global crawler?

[faner11]

app/crawlee:start: /Users/ha/Codes/arcteryx-crawlee/node_modules/.pnpm/@crawlee[email protected]/node_modules/@crawlee/memory-storage/fs/key-value-store/fs.js:72
@app/crawlee:start: throw new Error(Could not find file at ${this.filePath});
@app/crawlee:start: ^
@app/crawlee:start:
@app/crawlee:start: Error: Could not find file at /Users/ha/Codes/arcteryx-crawlee/apps/crawlee/storage/key_value_stores/default/SDK_my-list-REQUEST_LIST_REQUESTS.bin
@app/crawlee:start: at KeyValueFileSystemEntry.get (/Users/ha/Codes/arcteryx-crawlee/node_modules/.pnpm/@crawlee[email protected]/node_modules/@crawlee/src/fs/key-value-store/fs.ts:51:23)
@app/crawlee:start: at KeyValueStoreClient.getRecord (/Users/ha/Codes/arcteryx-crawlee/node_modules/.pnpm/@crawlee[email protected]/node_modules/@crawlee/src/resource-clients/key-value-store.ts:224:23)
@app/crawlee:start: at KeyValueStore.getValue (/Users/ha/Codes/arcteryx-crawlee/node_modules/.pnpm/@crawlee[email protected]/node_modules/@crawlee/src/storages/key_value_store.ts:227:24)
@app/crawlee:start: at RequestList._getPersistedState (/Users/ha/Codes/arcteryx-crawlee/node_modules/.pnpm/@crawlee[email protected]/node_modules/@crawlee/src/storages/request_list.ts:758:23)
@app/crawlee:start: at RequestList._loadStateAndPersistedRequests (/Users/ha/Codes/arcteryx-crawlee/node_modules/.pnpm/@crawlee[email protected]/node_modules/@crawlee/src/storages/request_list.ts:621:33)
@app/crawlee:start: at RequestList.initialize (/Users/ha/Codes/arcteryx-crawlee/node_modules/.pnpm/@crawlee[email protected]/node_modules/@crawlee/src/storages/request_list.ts:426:44)
@app/crawlee:start: at Function.open (/Users/ha/Codes/arcteryx-crawlee/node_modules/.pnpm/@crawlee[email protected]/node_modules/@crawlee/src/storages/request_list.ts:963:9)
@app/crawlee:start: at createCrawler (/Users/ha/Codes/arcteryx-crawlee/apps/crawlee/src/main.ts:44:23)
@app/crawlee:start: at taskRun (/Users/ha/Codes/arcteryx-crawlee/apps/crawlee/src/main.ts:90:19)
@app/crawlee:start: at Job.job (/Users/ha/Codes/arcteryx-crawlee/apps/crawlee/src/main.ts:123:5)
@app/crawlee:start: Emitted 'error' event on Job instance at:
@app/crawlee:start: at /Users/ha/Codes/arcteryx-crawlee/node_modules/.pnpm/[email protected]/node_modules/node-schedule/lib/Invocation.js:277:17
@app/crawlee:start:
@app/crawlee:start: Node.js v20.12.2

[faner11]

@B4nan Bro, repeating requests makes sense, for example, I want to know if a certain product is restocked. I plan to request every five minutes. What should I do in this scene

[B4nan]

I don't understand what you mean. I said you should not duplicate support requests in GH issues, I haven't said a thing about what you are asking about.

[faner11]

Should we continue discussing here now or should I open a new issue according to the standard

[B4nan]

Your problem is most likely the manual storage purge, both of your crawler instances use the same default storages, but you wipe them by hand with the purgeDefaultStorages call.

For this use case, I would suggest you disable storage persistence completely:

import { Configuration, PlaywrightCrawler, ProxyConfiguration } from 'crawlee'
import { scheduleJob } from 'node-schedule'

import { router } from './routes'

const createCrawler = () => {
  const config = new Configuration({ persistStorage: false }) // <== added `persistStorage: false` here
  const crawler = new PlaywrightCrawler(
    {
      proxyConfiguration: new ProxyConfiguration({ proxyUrls: [p] }),
      requestHandler: router,
      maxRequestsPerCrawl: 50,
      maxRequestRetries: 0,
    },
    config,
  )
  return crawler
}

const taskRun = async () => {
  const crawler = createCrawler()
  const linkList = ['https://github.com/xx1', 'https://github.com/xx2']
  await crawler.run(linkList).finally(() => {
//    void purgeDefaultStorages() // <== now you don't need this
  })
}

scheduleJob('* 0-4 * * *', async () => {
  await taskRun()
})

scheduleJob('*/2 5-10 * * *', async () => {
  await taskRun()
})

I would keep separate crawler instances here, since you might run them both at the same time, you can't really share the state of them.

[faner11]

Thank you, I will try it today and give you feedback

[faner11]

Using this method, the problem is solved, however a new problem is encountered, after running for a few hours, it starts reporting out of memory, can we manually free the memory. If we use sub-threads, each time in the task is executed in the sub-thread, after execution release the sub-thread can we alleviate this problem.
I've searched for some relevant discussions that

• Memory overloaded: overestimated used memory #1722
• Proxy config seems to be causing problems related to memory leak #2556

@app/crawlee:start: WARN  PlaywrightCrawler:AutoscaledPool:Snapshotter: Memory is critically overloaded. Using 5378 MB of 4096 MB (131%). Consider increasing available memory.
@app/crawlee:start: WARN  PlaywrightCrawler: Reclaiming failed request back to the list or queue. Detected a session error, rotating session... 
@app/crawlee:start: page.goto: net::ERR_TUNNEL_CONNECTION_FAILED at

@app/crawlee:start: <--- Last few GCs --->
@app/crawlee:start: 
@app/crawlee:start: [84866:0x130008000] 22473035 ms: Scavenge 4009.9 (4138.5) -> 4007.5 (4138.5) MB, 12.62 / 0.04 ms  (average mu = 0.889, current mu = 0.903) allocation failure; 
@app/crawlee:start: [84866:0x130008000] 22473044 ms: Scavenge 4010.2 (4138.5) -> 4008.7 (4138.5) MB, 4.25 / 0.00 ms  (average mu = 0.889, current mu = 0.903) task; 
@app/crawlee:start: [84866:0x130008000] 22473069 ms: Scavenge 4010.7 (4138.5) -> 4008.0 (4146.5) MB, 9.08 / 0.00 ms  (average mu = 0.889, current mu = 0.903) task; 
@app/crawlee:start: 
@app/crawlee:start: 
@app/crawlee:start: <--- JS stacktrace --->
@app/crawlee:start: 
@app/crawlee:start: FATAL ERROR: Reached heap limit Allocation failed - JavaScript heap out of memory
@app/crawlee:start: ----- Native stack trace -----
@app/crawlee:start: 
@app/crawlee:start:  1: 0x1048f90e8 node::OOMErrorHandler(char const*, v8::OOMDetails const&) [/Users/ha/Library/pnpm/nodejs/20.12.2/bin/node]
@app/crawlee:start:  2: 0x104a7f120 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, v8::OOMDetails const&) [/Users/ha/Library/pnpm/nodejs/20.12.2/bin/node]
@app/crawlee:start:  3: 0x104c53668 v8::internal::Heap::GarbageCollectionReasonToString(v8::internal::GarbageCollectionReason) [/Users/ha/Library/pnpm/nodejs/20.12.2/bin/node]
@app/crawlee:start:  4: 0x104c52144 v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [/Users/ha/Library/pnpm/nodejs/20.12.2/bin/node]
@app/crawlee:start:  5: 0x104cb8f50 v8::internal::MinorGCJob::Task::RunInternal() [/Users/ha/Library/pnpm/nodejs/20.12.2/bin/node]
@app/crawlee:start:  6: 0x104959dd8 node::PerIsolatePlatformData::RunForegroundTask(std::__1::unique_ptr<v8::Task, std::__1::default_delete<v8::Task>>) [/Users/ha/Library/pnpm/nodejs/20.12.2/bin/node]
@app/crawlee:start:  7: 0x104958b30 node::PerIsolatePlatformData::FlushForegroundTasksInternal() [/Users/ha/Library/pnpm/nodejs/20.12.2/bin/node]
@app/crawlee:start:  8: 0x1052ca274 uv__async_io [/Users/ha/Library/pnpm/nodejs/20.12.2/bin/node]
@app/crawlee:start:  9: 0x1052dc34c uv__io_poll [/Users/ha/Library/pnpm/nodejs/20.12.2/bin/node]
@app/crawlee:start: 10: 0x1052ca838 uv_run [/Users/ha/Library/pnpm/nodejs/20.12.2/bin/node]
@app/crawlee:start: 11: 0x10481d6f0 node::SpinEventLoopInternal(node::Environment*) [/Users/ha/Library/pnpm/nodejs/20.12.2/bin/node]
@app/crawlee:start: 12: 0x1049389e4 node::NodeMainInstance::Run(node::ExitCode*, node::Environment*) [/Users/ha/Library/pnpm/nodejs/20.12.2/bin/node]
@app/crawlee:start: 13: 0x104938764 node::NodeMainInstance::Run() [/Users/ha/Library/pnpm/nodejs/20.12.2/bin/node]
@app/crawlee:start: 14: 0x1048b7d58 node::Start(int, char**) [/Users/ha/Library/pnpm/nodejs/20.12.2/bin/node]
@app/crawlee:start: 15: 0x1806c10e0 start [/usr/lib/dyld]

[faner11]

Each additional instance of crawlee created adds about 300M of memory, which is not freed, but I solved this problem by executing it in child_process.