Error: Invalid "proxyUrl" option: only HTTP proxies are currently supported

[mehrdad-shokri]

Which package is this bug report for? If unsure which one to select, leave blank

@crawlee/playwright (PlaywrightCrawler)

Issue description

The docs never mention that only http proxies are supported. I think using http proxies are a security risk. Digging deeper you end up in here which crawlee uses. I think it should support HTTPS proxies as well.

Code sample

const proxyConfiguration = new ProxyConfiguration({
    proxyUrls: [
      'http://Username:Password@proxyUrl:PORT',
    ],
  });
  const crawler = new PlaywrightCrawler(
    {
      proxyConfiguration,
      // Use the requestHandler to process each of the crawled pages.
      async requestHandler({request, page, enqueueLinks, log, crawler}) {
        const title = await page.title();
        content = await page.content();
        log.info(`Title of ${request.loadedUrl} is '${title}'`);
        // Save results as JSON to ./storage/datasets/default
        await Dataset.pushData({title, url: request.loadedUrl, content});

        // Extract links from the current page
        // and add them to the crawling queue.
        await enqueueLinks();
      },
      maxRequestsPerCrawl: 1,
      maxConcurrency: 20,
      retryOnBlocked: true,
      maxRequestRetries: 10,
    },
    new Configuration({
      persistStorage: false,
      maxUsedCpuRatio: 0.95,
      availableMemoryRatio: 0.5,
    }),
  );

await crawler.run([url])

Package version

[email protected] [email protected]

Node.js version

v20.10.0 [email protected]

Operating system

macOS

Apify platform

• Tick me if you encountered this issue on the Apify platform

I have tested this on the next release

No response

Other context

No response

[barjin]

Hello - and thank you for your interest in this project.

Can you please provide reproduction scenario for the issue you are having?

"I think using http proxies are a security risk"

Note that this is not true - if you are connecting to the target server via HTTPS, the traffic is still end-to-end encrypted. With HTTP proxies, this is achieved via HTTP CONNECT method, which creates an opaque data tunnel from the client to the proxy server, through which the encrypted data is transferred. The intermediate proxy server cannot read this data (as it's encrypted).

If you are connecting to an HTTP target server (or you decide to fiddle around with the TLS settings - see e.g. comments under this issue), the proxy can indeed act as MITM and read your traffic - but you really have to want this - it will never happen with the default

[mehrdad-shokri]

Hello,

Note that this is not true - if you are connecting to the target server via HTTPS, the traffic is still end-to-end encrypted. With HTTP proxies, this is achieved via HTTP CONNECT method, which creates an opaque data tunnel from the client to the proxy server, through which the encrypted data is transferred. The intermediate proxy server cannot read this data (as it's encrypted).

Is it true for every kind of residential proxy provider or you're talking about Apify proxies. So if I connect to an http residential proxy(no matter the provider) they wouldn't be able to read the contents of the request? I'm aware that at least they monitor the host name and links that I'm scrapping. So wondering if they also can see the request payload as well.