Streaming data

[geezz07]

To give a little context I am trying to stream my scraped data back to the user via API endpoint. I'm running into an issue where I pass the WriteableStream to the userData and within the route handlers I am writing the data to the stream. The issue is the userData does not maintain the properties and functions of my object being passed in, "ERROR CheerioCrawler: Request failed and reached maximum retries. TypeError: stream.write is not a function"

const { readable, writable } = new TransformStream();
const writer = writable.getWriter();

async function startCluster(writer: WritableStreamDefaultWriter<string>) {
      const crawler = new CheerioCrawler(
            Object.assign(
                { requestQueue: await RequestQueue.open(CRAWLERS.UNIQUE_KEY) },
                crawlOptions
            ),
            new Configuration({
                defaultKeyValueStoreId: CRAWLERS.UNIQUE_KEY,
                defaultDatasetId: CRAWLERS.UNIQUE_KEY,
                defaultRequestQueueId: CRAWLERS.UNIQUE_KEY
            }));

      const crawl1 = crawler.run([
            {
                url: `https://www.abc.com`,
                userData: {
                    stream: writer
                }
            },
        ]);
      await crawl1;
}

export const router = createCheerioRouter();

router.addDefaultHandler(async ({ request, enqueueLinks, log }) => {
    await enqueueLinks({
        selector: 'details-link',
        label: 'DETAIL',
        userData: request.userData
    });
});

router.addHandler('DETAIL', async ({ request, $, log, pushData }) => {
    ...

    await pushData(product);

    const stream = request?.userData?.stream as WritableStreamDefaultWriter<string>;
    if (stream) {
        await stream.write(JSON.stringify(product)); // <---- This is where I run into a problem. 
        //"ERROR CheerioCrawler: Request failed and reached maximum retries. TypeError: stream.write is not a function"
    }
});

startCrawler(writer);

[B4nan]

userData needs to be serializable, technically you can't put anything else than a JSON in there.

Maybe have a map of request.uniqueKey to your stream handles instead of adding it there?

[B4nan]

I don't have time to write you a complete example, please show me what you tried first.

[B4nan]

Uh sorry, I see you already have the example code in the OP :]

[B4nan]

Hmm looking at your code I am honestly not sure why are you using adding the stream to the user data? You got one stream per crawler instance, right?

[geezz07]

It is the same stream object being passed to multiple crawlers via userData. Wasn't sure how else to pass the stream object to the routers. I'm now trying something like this...

let _router: RouterHandler<CheerioCrawlingContext<any, any>>;

export const initRouter = (writer: WritableStreamDefaultWriter<string>) => {
    if (_router== undefined) {
        _router= createCheerioRouter();
    }

    // Metro, Food Basics
    _router.addHandler('DETAIL', async ({ request, $, log, pushData }) => {
        ...

        if (writer) {
            await writer.write(JSON.stringify(product));
        }
    });


    // This is a fallback route which will handle the start URL
    // as well as the LIST labeled URLs.
    _router.addDefaultHandler(async ({ request, enqueueLinks, log }) => {
        ...
    });

   return _router;
}

UPDATE: Tried it and this is working for me