HTTP Crawler Memory Leak?

[ikeg225]

Which package is this bug report for? If unsure which one to select, leave blank

@crawlee/http (HttpCrawler)

Issue description

I'm not completely comfortable with Crawlee and dealing with memory leaks in NodeJS with Express, but I suspect there might be a memory leak issue with Crawlee. I have been running it on my server for the past few days, handling tens of thousands of requests each day. However, I keep encountering memory issues. Each time I check the memory usage, it is really high and continues to grow.

Below is some simple code using the HTTP crawler. I might not be properly disposing of the Crawlee instance, which could be causing the memory to increase with every run. The attached picture shows the growth of the process—each snapshot represents another 100 individual runs. The increase is minimal each time, but it adds up significantly over time.

Snapshot 3 - 0 runs, just initiating the server
Snapshot 4 - 100 runs, since the start
Snapshot 5 - 200 runs, since the start
Snapshot 6 - 300 runs, since the start

I let the crawler sit for a few minutes after finishing each 100 runs to let the memory settle.

This is the code I ran to test the express server:

for _ in range(100):
  req = requests.post("http://localhost:3002", json={
      "requests": [
          {"url": "https://httpbin.org/get"}, 
      ]}
  )

Please let me know if I am doing something wrong or this is how it's supposed to be! Again, I am not too comfortable with all of this so not really sure, but all I know is that my server keeps running out of memory which is 4GB.

Also I did a lot of searching for previous memory leak issues with crawlee and came across a few threads. I believe I implemented what was suggested, sorry if I missed anything!

Code sample

import express from "express";
import type { RequestOptions } from "@crawlee/core";
import { HttpCrawler, Configuration } from "crawlee";

const app = express();
const port = 3002;

app.use(express.json());

type Request = {
  apiKey: string;
  requests: RequestOptions[];
};

type Response = {
  url: string;
  method: string;
  payload?: string;
  headers?: Record<string, string>;
  content: string;
};

function updateRequests(requests: Partial<Request>) {
  requests.requests?.forEach((request) => {
    if (request.method === "POST") {
      request.useExtendedUniqueKey = true;
    }
  });
  return requests.requests;
}

app.post("/", async (req, res) => {
  try {
    const request = req.body as Partial<Request>;

    const responses: Response[] = [];

    const crawler = new HttpCrawler(
      {
        maxRequestRetries: 50,
        useSessionPool: false,
        persistCookiesPerSession: false,
        async requestHandler({ request, body }) {
          const response = {
            url: request.url,
            method: request.method,
            payload: request.payload,
            headers: request.headers,
            content: body.toString(),
          };

          responses.push(response);
        },
      },
      new Configuration({
        persistStorage: false,
      })
    );

    await crawler.run(updateRequests(request));


    await crawler.teardown();

    return res.send(responses);
  } catch (error: any) {
    console.log(error);
    return res.status(500).send(error.message);
  }
});

app.listen(port, () => {
  console.log(`Server is running on port ${port}`);
});

Package version

3.10

Node.js version

20.11.1

Operating system

No response

Apify platform

• Tick me if you encountered this issue on the Apify platform

I have tested this on the next release

No response

Other context

No response

[barjin]

Hello and thank you for your interest in Crawlee!

Unfortunately, with your scenario, I wasn't able to reproduce the error. The increase from ~30 to ~40 MB in the snapshots you've made is completely normal and cannot be considered indicative of any problem. The mark-sweep garbage collection is an expensive operation, so Node.JS always tries to defer it - if you have 4GB of RAM available, Node won't run the garbage collection because of 10 extra megabytes.

You can try using node-oom-heapdump which takes a heap snapshot right when Node encounters the OOM error. This might give you a better idea about what's going on in your process. If your process doesn't fail with an OOM error (and the only issue is that the process is eating more memory than you would expect), then it's probably just Node utilizing all the memory you give it. You can change this using the --max-old-space-size CLI option for Node.