ie8/9 fails under sauce connect

[joecomotion]

I'm trying to use yeti for running mail unit tests on saucelabs. At home, if I poke a hole in my firewall, I can run against ie9 and ie10 no problem. When I use sauce connect, though (which is how we'll have to run it at Yahoo), ie10 works, but ie9 does not.

When running w/ sauce connect, here's the command I'm using for ie9:
yeti --wd-url "http://:@ondemand.saucelabs.com" --browser "windows 7/ie/9" tests/unit/testAttachments.html

In the failing case, I see entries in my sauce connect log that look like this:

2013-11-08 11:33:50.613:INFO::Started [email protected]:4445
2013-11-08 11:33:50,614 - Selenium HTTP proxy listening on port 4445
2013-11-08 11:33:50,746 - Successful handshake with Sauce Connect server
2013-11-08 11:33:50,759 - Tunnel host version: 0.1.0, remote endpoint ID: 7518c05522c24423974fe166adca20a2
2013-11-08 11:33:50,762 - Connected! You may start your tests.
2013-11-08 11:34:18,065 - Request started: GET http://192.168.1.133:9000/agent/138393924933610484146
2013-11-08 11:34:18,079 - GET http://192.168.1.133:9000/agent/138393924933610484146 -> 200 (14ms, 973 bytes)
2013-11-08 11:34:18,140 - Request started: GET http://192.168.1.133:9000/agent/public/yeti.css
2013-11-08 11:34:18,153 - GET http://192.168.1.133:9000/agent/public/yeti.css -> 200 (13ms, 1691 bytes)
2013-11-08 11:34:18,181 - Request started: GET http://192.168.1.133:9000/agent/public/capture.js
2013-11-08 11:34:18,187 - GET http://192.168.1.133:9000/agent/public/capture.js -> 200 (7ms, 169741 bytes)
2013-11-08 11:34:22,619 - Request started: GET http://192.168.1.133:9000/favicon.ico
2013-11-08 11:34:22,625 - GET http://192.168.1.133:9000/favicon.ico -> 404 (5ms, 49 bytes)
2013-11-08 11:34:22,648 - Request started: GET http://192.168.1.133:9000/tower/info?t=1383939249428
2013-11-08 11:34:22,654 - GET http://192.168.1.133:9000/tower/info?t=1383939249428 -> 200 (7ms, 78 bytes)
2013-11-08 11:34:22,742 - Request started: POST http://192.168.1.133:9000/tower/301/tngm7itr/xhr_streaming?t=1383939249524
2013-11-08 11:34:23,375 - Request started: POST http://192.168.1.133:9000/tower/301/mh901pel/xhr?t=1383939250117
2013-11-08 11:34:23,380 - POST http://192.168.1.133:9000/tower/301/mh901pel/xhr?t=1383939250117 -> 200 (6ms, 2 bytes)
2013-11-08 11:34:23,513 - Request started: POST http://192.168.1.133:9000/tower/301/mh901pel/xhr?t=1383939250296
2013-11-08 11:34:23,517 - POST http://192.168.1.133:9000/tower/301/mh901pel/xhr?t=1383939250296 -> 200 (4ms, 31 bytes)
2013-11-08 11:34:23,624 - Request started: POST http://192.168.1.133:9000/tower/301/mh901pel/xhr_send?t=1383939250403
2013-11-08 11:34:23,631 - POST http://192.168.1.133:9000/tower/301/mh901pel/xhr_send?t=1383939250403 -> 500 (6ms, 17 bytes)
2013-11-08 11:34:23,642 - Request started: POST http://192.168.1.133:9000/tower/301/mh901pel/xhr?t=1383939250404
2013-11-08 11:34:28,703 - Request started: GET http://192.168.1.133:9000/tower/info?t=1383939255477
2013-11-08 11:34:28,707 - GET http://192.168.1.133:9000/tower/info?t=1383939255477 -> 200 (4ms, 79 bytes)
2013-11-08 11:34:28,757 - Request started: POST http://192.168.1.133:9000/tower/266/rod1bpp0/xhr_streaming?t=1383939255532
2013-11-08 11:34:29,286 - Request started: POST http://192.168.1.133:9000/tower/266/69w0mm7y/xhr?t=1383939256048
2013-11-08 11:34:29,289 - POST http://192.168.1.133:9000/tower/266/69w0mm7y/xhr?t=1383939256048 -> 200 (4ms, 2 bytes)
2013-11-08 11:34:29,344 - Request started: POST http://192.168.1.133:9000/tower/266/69w0mm7y/xhr?t=1383939256118
2013-11-08 11:34:29,348 - POST http://192.168.1.133:9000/tower/266/69w0mm7y/xhr?t=1383939256118 -> 200 (5ms, 31 bytes)
2013-11-08 11:34:29,470 - Request started: POST http://192.168.1.133:9000/tower/266/69w0mm7y/xhr_send?t=1383939256242
2013-11-08 11:34:29,473 - POST http://192.168.1.133:9000/tower/266/69w0mm7y/xhr_send?t=1383939256242 -> 500 (3ms, 17 bytes)
2013-11-08 11:34:29,489 - Request started: POST http://192.168.1.133:9000/tower/266/69w0mm7y/xhr?t=1383939256243
2013-11-08 11:34:34,572 - Request started: GET http://192.168.1.133:9000/tower/info?t=1383939261349
2013-11-08 11:34:34,575 - GET http://192.168.1.133:9000/tower/info?t=1383939261349 -> 200 (3ms, 78 bytes)

[joecomotion]

On the yeti command line in the failure case, I never see a browser connect, e.g. stuck at:

Waiting for Hub to launch browsers...

[reid]

Could you share a successful log from Sauce Connect for IE 10? I wonder if the problem is related to a SockJS transport in older IE (like xhr-streaming) having problem's with Sauce's proxy.

On Nov 25, 2013, at 4:00 PM, joecomotion [email protected] wrote:

On the yeti command line in the failure case, I never see a browser connect, e.g. stuck at:

Waiting for Hub to launch browsers...

—
Reply to this email directly or view it on GitHub.

[joecomotion]

Here's a successful IE10 run:

2013-11-08 11:30:18.915:INFO::Started [email protected]:52283
.---------------------------------------------------.
| Have questions or need help with Sauce Connect? |
| Contact us: http://support.saucelabs.com/forums |

| Terms of Service: http://saucelabs.com/tos |

2013-11-08 11:30:18,930 - / Starting
2013-11-08 11:30:18,934 - Please wait for "You may start your tests" to start your tests.
2013-11-08 11:30:18,943 - Forwarding: ['sauce-connect.proxy']:['80'] -> 127.0.0.1:['52283']
2013-11-08 11:30:18,951 - Succesfully connected to local server 127.0.0.1:52283 in 6ms
2013-11-08 11:30:20,042 - {"no_ssl_bump_domains":null,"squid_config":null,"use_caching_proxy":true,"metadata":{"PythonVersion":"2.5.1","OwnerHost":"127.0.0.1","Release":"3.0-r30","OwnerPorts":["52283"],"Ports":["80"],"Platform":"Java-1.7.0_40-Java_HotSpot-TM-_64-Bit_Server_VM,_24.0-b56,_Oracle_Corporation-on-Mac_OS_X-10.8.5-x86_64","Build":"46","ScriptRelease":46,"ScriptName":"sauce_connect"},"use_kgp":true,"tunnel_identifier":"","shared_tunnel":false,"fast_fail_regexps":null,"ssh_port":443,"direct_domains":null,"vm_version":"","domain_names":["sauce-connect.proxy"]}
2013-11-08 11:30:25,677 - Tunnel remote VM is provisioned (9c6658bee15248adaa8d25f1c157b0ee)
2013-11-08 11:30:26,341 - Tunnel remote VM is booting ..
2013-11-08 11:30:37,448 - Tunnel remote VM is running at maki8042.miso.saucelabs.com
2013-11-08 11:30:37,463 - Succesfully connected to local server 127.0.0.1:52283 in 0ms
2013-11-08 11:30:37,467 - Starting connection to tunnel host...
2013-11-08 11:30:37,467 - Connecting to tunnel host maki8042.miso.saucelabs.com as joecomotion
2013-11-08 11:30:37,598 - Forwarding Selenium with ephemeral port 52287
2013-11-08 11:30:37.601:INFO::jetty-7.x.y-SNAPSHOT
2013-11-08 11:30:37.602:INFO::Started [email protected]:4445
2013-11-08 11:30:37,602 - Selenium HTTP proxy listening on port 4445
2013-11-08 11:30:37,803 - Successful handshake with Sauce Connect server
2013-11-08 11:30:37,818 - Tunnel host version: 0.1.0, remote endpoint ID: 24efbaa998f54b0db2823c0421e7f2bd
2013-11-08 11:30:37,821 - Connected! You may start your tests.
2013-11-08 11:31:24,451 - Request started: GET http://192.168.1.133:9000/agent/13839390762307021871
2013-11-08 11:31:24,470 - GET http://192.168.1.133:9000/agent/13839390762307021871 -> 200 (19ms, 973 bytes)
2013-11-08 11:31:24,539 - Request started: GET http://192.168.1.133:9000/agent/public/yeti.css
2013-11-08 11:31:24,555 - GET http://192.168.1.133:9000/agent/public/yeti.css -> 200 (15ms, 1691 bytes)
2013-11-08 11:31:24,608 - Request started: GET http://192.168.1.133:9000/agent/public/capture.js
2013-11-08 11:31:24,615 - GET http://192.168.1.133:9000/agent/public/capture.js -> 200 (7ms, 169741 bytes)
2013-11-08 11:31:29,092 - Request started: GET http://192.168.1.133:9000/favicon.ico
2013-11-08 11:31:29,096 - GET http://192.168.1.133:9000/favicon.ico -> 404 (5ms, 49 bytes)
2013-11-08 11:31:29,115 - Request started: GET http://192.168.1.133:9000/tower/info?t=1383939075826
2013-11-08 11:31:29,119 - GET http://192.168.1.133:9000/tower/info?t=1383939075826 -> 200 (3ms, 78 bytes)
2013-11-08 11:31:29,232 - Request started: POST http://192.168.1.133:9000/tower/015/ncvnuot9/xhr_streaming?t=1383939075939
2013-11-08 11:31:29,796 - Request started: POST http://192.168.1.133:9000/tower/015/9mv5qq1j/xhr?t=1383939076487
2013-11-08 11:31:29,802 - POST http://192.168.1.133:9000/tower/015/9mv5qq1j/xhr?t=1383939076487 -> 200 (5ms, 2 bytes)
2013-11-08 11:31:29,871 - Request started: POST http://192.168.1.133:9000/tower/015/9mv5qq1j/xhr?t=1383939076581
2013-11-08 11:31:29,877 - POST http://192.168.1.133:9000/tower/015/9mv5qq1j/xhr?t=1383939076581 -> 200 (5ms, 31 bytes)
2013-11-08 11:31:29,977 - Request started: POST http://192.168.1.133:9000/tower/015/9mv5qq1j/xhr_send?t=1383939076683
2013-11-08 11:31:29,986 - POST http://192.168.1.133:9000/tower/015/9mv5qq1j/xhr_send?t=1383939076683 -> 204 (9ms, 0 bytes)
2013-11-08 11:31:30,000 - Request started: POST http://192.168.1.133:9000/tower/015/9mv5qq1j/xhr?t=1383939076684
2013-11-08 11:31:30,003 - POST http://192.168.1.133:9000/tower/015/9mv5qq1j/xhr?t=1383939076684 -> 200 (5ms, 192 bytes)
2013-11-08 11:31:30,098 - Request started: POST http://192.168.1.133:9000/tower/015/9mv5qq1j/xhr?t=1383939076807
2013-11-08 11:31:30,119 - Request started: GET http://192.168.1.133:9000/agent/13839390762307021871/batch/1383939076230353167/test/tests/unit/testAttachments.html
2013-11-08 11:31:30,125 - GET http://192.168.1.133:9000/agent/13839390762307021871/batch/1383939076230353167/test/tests/unit/testAttachments.html -> 200 (7ms, 530 bytes)
2013-11-08 11:31:30,207 - Request started: GET http://192.168.1.133:9000/agent/public/inject.js
2013-11-08 11:31:30,213 - GET http://192.168.1.133:9000/agent/public/inject.js -> 200 (7ms, 184957 bytes)
2013-11-08 11:31:30,240 - Request started: GET http://192.168.1.133:9000/agent/13839390762307021871/batch/1383939076230353167/test/release/js/utils.js

[ipeychev]

Is there any progress on this issue? We have exactly the same problem.

[reid]

@ipeychev @joecomotion I found the source of the problem. Thanks for your patience. This bug was difficult to track down.

---

Yeti uses SockJS to communicate with browsers. SockJS implements many protocols to accommodate old and new browsers in the fastest way.

• For IE 10, the websocket protocol is used.
• For IE 8 and 9, xdr-streaming is used. This does not work.
• For IE 7 and below, xhr-polling or jsonp-polling is used.

Sauce configures all browsers to use the Selenium HTTP proxy server to allow self-signed certificates and Sauce Connect to work. However, this proxy does not properly handle xdr-streaming or xhr-streaming requests from Yeti, which causes Yeti to hang when the proxy is in use. This may be because IE 8 and 9 require an unusual 2KB prefix for XDomainRequest responses due to a bug in the network wrapper below XDomainRequest. I'd wager that the Selenium proxy used by Sauce does not properly handle this prefix. Since the 2KB prefix is sent to all browsers, this prevents xhr-streaming and xdr-streaming protocols from working on Sauce with the proxy enabled.

Because the proxy causes problems, Yeti currently always enables the avoid-proxy capability when launching a browser.

So there's the problem. We can't simply add an option to omit avoid-proxy when using Sauce Connect, since it still wouldn't work: the proxy doesn't properly relay xhr-streaming and xdr-streaming requests.

---

We can disable the xdr-streaming and xhr-streaming protocol settings on the client when Sauce Connect is in use. This will cause slower alternatives to be used instead.

An even better option would be to detect the failure of the xdr-streaming protocol on the client-side and fall back automatically to xdr-polling or some other protocol. I'd prefer this option if at all possible. This may be possible to implement in Yeti's tempest.js but it'll be way better to include it upstream inside of sockjs-client.

I'll experiment with these options this week. @joecomotion, let me know if this fix remains important to you.

[joecomotion]

@reid This was for a project in Yahoo Mail. I am currently on sabbatical from Yahoo and so am a bit out of touch with the priority --- I will ping some folks and follow-up.

[ipeychev]

@reid This is important for us too. We use Yeti to run AlloyUI tests on SauceLabs, but currently we can only run them against IE7 and IE10.

[joecomotion]

Might be worth talking to someone at sauce. Maybe they can fix it on their side.

-j

Sent from my iPhone

On Feb 5, 2014, at 3:54 PM, Reid Burke [email protected] wrote:

@ipeychev @joecomotion I found the source of the problem. Thanks for your patience. This bug was difficult to track down.

Yeti uses SockJS to communicate with browsers. SockJS implements many protocols to accommodate old and new browsers in the fastest way.

For IE 10, the websocket protocol is used.
For IE 8 and 9, xdr-streaming is used. This does not work.
For IE 7 and below, xhr-polling or jsonp-polling is used.
Sauce configures all browsers to use the Selenium HTTP proxy server to allow self-signed certificates and Sauce Connect to work. However, this proxy does not properly handle xdr-streaming or xhr-streaming requests from Yeti, which causes Yeti to hang when the proxy is in use. This may be because IE 8 and 9 require an unusual 2KB prefix for XDomainRequest responses due to a bug in the network wrapper below XDomainRequest. I'd wager that the Selenium proxy used by Sauce does not properly handle this prefix. Since the 2KB prefi x is sent to all browsers, this prevents xhr-streaming and xdr-streaming protocols from working on Sauce with the proxy enabled.

Because the proxy causes problems, Yeti currently always enables the avoid-proxy capability when launching a browser.

So there's the problem. We can't simply add an option to omit avoid-proxy when using Sauce Connect, since it still wouldn't work: the proxy doesn't properly relay xhr-streaming and xdr-streaming requests.

We can disable the xdr-streaming and xhr-streaming protocol settings on the client when Sauce Connect is in use. This will cause slower alternatives to be used instead.

An even better option would be to detect the failure of the xdr-streaming protocol on the client-side and fall back automatically to xdr-polling or some other protocol. I'd prefer this option if at all possible. This may be possible to implement in Yeti's tempest.js but it'll be way better to include it upstream inside of sockjs-client.

I'll experiment with these options this week. @joecomotion, let me know if this fix remains important to you.

—
Reply to this email directly or view it on GitHub.

[JetFault]

Important for YMail also :)

[ipeychev]

Following what @joecomotion suggested, I contacted SauceLabs and tolday they told me they "opened an internal ticket to investigate this".

[cpanato]

cool!!

[ldx]

Hey, sorry for reviving an old thread, let me know if this is outdated now.

As for setting avoid-proxy to true, that will only disable the built-in Selenium server. The proxy used by Sauce Connect internally should still be respected.

If you guys can give me a Sauce job id, I can look into what the problem is.

[DylanLacey]

@ldx I'll send you one internally (Sorry to backchannel everyone! Don't want to send around client details).