Useragent not changed by Crawlee

[galaczi]

Which package is this bug report for? If unsure which one to select, leave blank

@crawlee/playwright (PlaywrightCrawler)

Issue description

Create a basic Playwright project and visit this website: https://www.whatismybrowser.com/detect/what-is-my-user-agent/

The useragent will be the same as if you check within your actual browser.

Code sample

const crawler = new PlaywrightCrawler({
    async requestHandler() {
        await sleep(10000)
    },
    headless: false,
});

await crawler.run(['https://www.whatismybrowser.com/detect/what-is-my-user-agent//']);

Package version

[email protected], [email protected]

Node.js version

v19.0.0

Operating system

Ubuntu

Apify platform

• Tick me if you encountered this issue on the Apify platform

I have tested this on the next release

No response

Other context

Crawlee should set a useragent header to avoid detection, as confirmed here:

https://crawlee.dev/api/browser-crawler/interface/BrowserLaunchContext#userAgent

I believe session pools also (should) handle useragents, but I am not 100% on that.

[B4nan]

They are not the same, at least not with the latest version:

Left is the one opened by crawlee, right is my regular chrome browser.

Keep in mind that what crawlee (or better say fingerprint-suite) generates for UA tries to be as close to a real UA as possible, so seeing something very similar (or even the same) is rather expected, not a bug.

[email protected], [email protected]

v3.1.4 is pretty old, you should always upgrade before you report anything. There might be fixes not just to crawlee, but to its transitive dependencies.

I believe session pools also (should) handle useragents, but I am not 100% on that.

Yes, that's what we want in the long run (we plan on calling that user pool instead), but the refactoring will be more complex, we'd like to have that for v4 (sometimes around the end of 2023 probably).