How do I detect response encoding and use it in crawlee?

[Yaekiou]

How to do something like this in crawlee?

// https://www.npmjs.com/package/chardet
import chardet from "chardet";

const response = await fetch("https://crawlee.dev/");
const arrayBuffer = await response.arrayBuffer();

const encoding = chardet.detect(Buffer.from(arrayBuffer))!;
const html = new TextDecoder(encoding).decode(arrayBuffer);

[metalwarrior665]

Crawlee doesn't use fetch so the API will be a bit different.

Either you can use BasicCrawler and just do the fetch yourself or you can use CheerioCrawler and then use the body property in the context to detect the encoding. For Cheerio, you might also need to expand allowedMimeTypes

[Yaekiou]

Thanks for your response. I decided to set the forceResponseEncoding to fix the encoding issue.