Home

The Mozilla Remote Reftest Extension

This is a Firefox extension which allows someone to run Mozilla reftests against tests hosted on a remote server. This can be useful when testing Firefox on devices which, due to various limitations, don’t allow reftest to operate in its traditional localhost mode.

Running remote reftests

First, you’ll need a copy of Mozilla’s JS-based HTTP server (httpd.js) running on a machine to host the reftests you want to run. The version of httpd.js which is in mozilla-central won’t quite do, as that version is hardcoded to only serve files from http://localhost, and it doesn’t know about some of the file extensions which are relevant to reftest (such as the reftest manifest .list files). I’ve included a modified version which works with remote reftest in the “httpd” directory. Copy the files to the directory containing the reftests you wish to serve, then launch it with this command-line:

/path/to/xpcshell -v 170 -f httpd.js server.js [your.ip.address]

If you omit your.ip.address, the server will only serve files from http://localhost:8888/, which isn’t very useful for running reftests remotely. Otherwise, the files will be served from http://your.ip.address:8888/

Next, install remotereftest.xpi in Firefox on the test machine, and launch the tool by choosing “Remote Reftest Tool” from Firefox’s Tools menu. An HTML page will appear with the reftest options:

1. manifest url – enter the http:// url of the reftest manifest .list file you’d like to run
2. starting test – enter the test number you want to start at; this is useful in case your test run always crashes at test X
3. chunk size – enter the number of tests you’d like to run per sequence; the reftest test window is closed and then reopened between sequences. It appears to be necessary to “chunk” the test run like this on devices with limited memory, in order to avoid specious crashes. Firefox on desktop OS’s can run all the tests in a single sequence.

After you start the test, you can see the last 10 lines of the log in the window, as well as a summary of results. The complete results are written to a file called results.log in the directory where the extension is located.

Development notes

• The extension currently doesn’t check for all profile configurations which could cause it to fail (e.g., disabling images).
• Logging of canvas images for failed tests (for use in the reftest analyzer) is currently disabled, as logging long strings causes some devices to crash.
• Test layout/reftests/object/image-no-useful-extension-typesniff.html currently fails because the httpd.js server returns files with no extension as application/octet-stream, whereas this test needs a file returned with no content type.

Bugs

While working on this extension I encountered a curious problem. Three (and only three, out of >3700 total tests) fail, because they were rendered incorrectly while reftest was running, but would be rendered correctly if I loaded the test files into the normal browser window. After much puzzlement and investigation, I discovered I could fix this problem by changing a small bit of code that opens the window where the reftesting takes place.

Original code:

      var wwatch = CC["@mozilla.org/embedcomp/window-watcher;1"]
                   .getService(Components.interfaces.nsIWindowWatcher);
      wwatch.openWindow(null,
         "chrome://remote-reftest/content/reftest.xul",
        "_blank", "chrome,dialog=no,all", args);

Here, args is an nsISupportsString which was originally used to pass the manifest url to the reftest code. The remote reftest extension performs data exchange differently and no longer needed this argument, so I had replaced args with null. Doing so caused the following three tests to break:

1. layout/reftests/bugs/371561-1.html
2. layout/reftests/bugs/467084-1.html
3. layout/reftests/svg/text-scale-01.svg

In each case, the failed tests was really being rendered incorrectly. Switching null back to args fixed the problem. Since I wasn’t using the last parameter, args could be any nsISupportsString, including an empty one, it just couldn’t be null. I’ve filed bug 513377 for this.