Using a proxy with OpenWPM

[palmeida]

Is it possible to use Selenium's desired_capabilities with OpenWPM? The goal is to configure a proxy for browsing before doing a crawl, so any alternative methods to achieve that goal would be helpful.

Thank you.

[vringar]

Hey @palmeida, we are currently exposing BrowserParams.prefs which should be enough to implement this.
Alternatively, you could configure the proxy in a profile by hand and then use that as a BrowserParams.seed_tar.

[palmeida]

Thanks, I'll try that. In the mean time, I did get it to work by editing ./openwpm/deploy_browsers/deploy_firefox.py and adding the code to use the proxy there. Of course it would be more flexible to use one of the alternatives you suggest, so I'll look into it.

Also, the way I did it was by creating a webdriver.DesiredCapabilities.FIREFOX['proxy'] dictionary with the proxy parameters and then setting that as the proxy option in fo, with set_capability, but that raised an error in ./openwpm/config.py (trying to resolve a str that was supposed to be a path). I have no idea how that error arises, but I had to catch and dismiss it, which is not ideal.

[palmeida]

I tried the BrowserParams.prefs approach, but I couldn't get it to work. Maybe I'm not configuring all the necessary settings. I just added this in demo.py, after all the other parameters are set:

browser_param.prefs = {
            "network.proxy.type": 1,
            "network.proxy.http": "127.0.0.1",
            "network.proxy.http_port": "24000",
            "network.proxy.ssl": "127.0.0.1",
            "network.proxy.ssl_port": "24000"
            }

Everything runs normally, and the parameters are being set at the beginning of the run, but it's not using the proxy.

[palmeida]

In case it's useful for anyone who stumbles into this, it's working now. The http_port and ssl_port in the prefs must be integers, not strings.

[vringar]

Great to hear that this worked out!
Thanks for sharing your solution