How to have a new different instance for every request so that users only get their data?

[onurusluca]

Say I am sending a request to the crawler, and it crawls the website and send the logs and data to the user.
When I send 2 or more requests, it mixes up the data and sends all users data to every user. It just adds the request the to the queue.

How to isolate the crawler so that it only serves to one request?
I have checked with headless: false, it creates a new instance for every request but it still processes all requests from all users on all instances with no idea which user has what data.

Maybe I need a way to compare userId from the startCrawlHandler and the one inside requestHandler and compare them?
Or a way to add requests to the queue and process them one by one?

I have tried with RequestQueue but I think it is something different than what I am talking? It didn't do any difference.

 const queueId = `crawl-session-${userId}-${projectId}`;

    const requestQueue = new RequestQueue({
        id: queueId,
    });
    const crawler = new PlaywrightCrawler({
    ...
   requestQueue, 
    });

Current code:
server:

...

app.post("/start-crawl", async (request, reply) => {
    console.log("[server]: Starting the crawl...");

    const { userId, projectId, startUrl } = request.body as any; // Cast to any to access properties directly

    // Check for missing parameters first
    if (!userId || !projectId || !startUrl) {
        const missingParam = ["userId", "projectId", "startUrl"].find(
            (param) => !request.body[param]
        );

        console.error(`[crawler]: Missing ${missingParam}`);
        return reply.status(400).send({
            message: `Missing ${missingParam} parameter.`,
        });
    }

    // Assuming startCrawlHandler is adapted for Fastify's request and reply objects
    await startCrawlHandler(request, reply);
});
...

crawler:

import { FastifyRequest, FastifyReply } from "fastify";
import { PlaywrightCrawler } from "crawlee";

import {
  createLogEventBroadcaster,
  cleanUpContent,
  takeScreenshotAndSave,
  saveCrawlData,
} from "./utils/index.js";
import type { CrawlRequestPayload, ExportData } from "./interfaces/index.js";
import { typesenseService } from "./helpers/feedToTypesense.js";

export const startCrawlHandler = async (
  request: FastifyRequest,
  reply: FastifyReply
): Promise<void> => {
  const {
    userId,
    applicationId,
    projectId,
    startUrl,
    takeScreenshot = true,
    colorScheme = "light",
    viewportSize = { width: 1024, height: 768 },
  } = request.body as CrawlRequestPayload;

  const broadcastLogEvent = createLogEventBroadcaster(
    userId,
    applicationId,
    projectId
  );

  broadcastLogEvent("init", "[crawler]: Starting the crawl", {
    userId,
    applicationId,
    projectId,
    startUrl,
    takeScreenshot,
    colorScheme,
    viewportSize,
  });

  // Acknowledge the crawl initiation
  reply.status(202).send({
    message: "Crawl started successfully.",
    userId,
    applicationId,
    projectId,
  });

  const crawler = new PlaywrightCrawler({
    headless: true,
    maxConcurrency: 100,
    maxRequestsPerMinute: 250,

    preNavigationHooks: [
      async ({ page }) => {
        await page.setViewportSize(viewportSize);
        await page.emulateMedia({ colorScheme });
      },
    ],
    browserPoolOptions: {
      useFingerprints: false,
    },

    // CRAWLING (runs for each page)
    async requestHandler({ request, page, log, enqueueLinks }) {
      log.info(`Processing the page: ${request.url}`);

      // Data extraction
      const title = await page.title();
      const content = await page.content();

      if (request.url.includes(startUrl)) {
        const structuredData: ExportData = {
          userId,
          applicationId,
          projectId,
          url: request.url,
          content: cleanUpContent(content),
          title,
          screenshotName: takeScreenshot
            ? await takeScreenshotAndSave(
                userId,
                applicationId,
                projectId,
                page
              )
            : "",

          crawledAt: new Date().toISOString(),
        };

        log.info(`Saving data to the database`, { projectId });

        broadcastLogEvent("crawled-page", "[crawler]: Page Crawled", {
          url: request.url,
          title: structuredData.title,
          screenshotName: structuredData.screenshotName,
          crawledAt: structuredData.crawledAt,
        });

        await saveCrawlData(structuredData);
      }
      await enqueueLinks();
    },
  });

  try {
    await crawler.run([{ url: startUrl }]);
    console.log("Crawl completed successfully");
  } catch (error: any) {
    console.log("Crawl failed:", error);
  }
};