diff --git a/README.md b/README.md index 32ed937..94eb084 100644 --- a/README.md +++ b/README.md @@ -1,36 +1,37 @@ -[![Build Status](https://github.com/jquery/testswarm/actions/workflows/CI.yaml/badge.svg?event=push)](https://github.com/jquery/testswarm/actions/workflows/CI.yaml) -[![Tested with QUnit](https://img.shields.io/badge/tested_with-qunit-9c3493.svg)](https://qunitjs.com/) +[![Tested with QUnit](https://qunitjs.com/testedwith.svg)](https://qunitjs.com/) -TestSwarm - Distributed Continuous Integration for JavaScript +TestSwarm ================= -TestSwarm provides distributed continuous integration testing for -JavaScript. +TestSwarm provides distributed continuous integration testing for JavaScript. -The main instance monitoring jQuery core and related projects runs at -[swarm.jquery.org](https://swarm.jquery.org/). +> **⚠️ Project status** +> TestSwarm remains used by jQuery Foundation projects such as jQuery and jQuery UI, but is no longer under active development. Critical issues may be patched, but new issues will not be addressed. +> +> We recommend reviewing these alternatives: [QTap](https://github.com/qunitjs/qtap), [Karma](https://karma-runner.github.io/), [Testem](https://github.com/testem/testem), [grunt-contrib-qunit](https://github.com/gruntjs/grunt-contrib-qunit), [browserstack-runner](https://github.com/browserstack/browserstack-runner/), [Airtap](https://github.com/airtap/airtap), [Intern](https://theintern.io/), [Web Test Runner](https://github.com/brandonaaron/web-test-runner-qunit). -## Project Status - -TestSwarm is used in projects of the jQuery Foundation, but it isn't under active development anymore. Although critical issues may be patched in the future, most open issues will remain unaddressed. +## Documentation -Within the jQuery Foundation, we're experimenting with alternative projects, to eventually shut down our own instance of TestSwarm: +* [About TestSwarm](./docs/About.md) (Philosophy, Architecture, How is it different?) +* [API Guide](./docs/API.md) +* [Automation Guide](./docs/Automation.md) +* [How to: Submit jobs](./scripts/addjob/README.md) +* [Project history](./docs/History.md) (Screenshots) -- [airtap](https://github.com/airtap/airtap) -- [Karma](https://karma-runner.github.io/) -- [Intern](https://theintern.io/) -- [browserstack-runner](https://github.com/browserstack/browserstack-runner/) +**Further reading**: -We recommend reviewing those and other alternatives. +* [JavaScript Testing Does Not Scale](http://ejohn.org/blog/javascript-testing-does-not-scale/), John Resig, 2009. +* [TestSwarm Alpha Open!](https://johnresig.com/blog/test-swarm-alpha-open/), John Resig, 2009. +* [JSConf talk: TestSwarm](http://ejohn.org/blog/jsconf-talk-games-performance-testswarm/), John Resig, 2009. +* [Video: TestSwarm Walkthrough](http://www.vimeo.com/6281121), John Resig, 2009. ## Quick start -Clone the repo, `git clone --recursive git://github.com/jquery/testswarm.git`. +Clone the repo -## Bug tracker - -Found a bug? Please report it using our [issue -tracker](https://github.com/jquery/testswarm/issues)! +```sh +git clone https://github.com/jquery/testswarm.git +``` ## Installation @@ -114,11 +115,6 @@ You're welcome to use the GitHub [issue tracker](https://github.com/jquery/tests Some of us are also on Gitter at [jquery/dev](https://gitter.im/jquery/dev). -## Documentation - -* [TestSwarm wiki](https://github.com/jquery/testswarm/wiki) -* [Submit jobs README](./scripts/addjob/README.md) - ## Copyright and license See [LICENSE.txt](./LICENSE.txt). @@ -134,10 +130,3 @@ Releases will be numbered in the following format: The `-alpha` suffix is used to indicate unreleased versions in development. For more information on SemVer, please visit . - -## History - -TestSwarm was originally created by [John Resig](https://johnresig.com/) as a -basic tool to support unit testing of the [jQuery JavaScript -library](https://jquery.com). It later become a [Mozilla Labs](https://labs.mozilla.com/) project, -and has since moved again to become a [jQuery Foundation](https://jquery.org/) project. diff --git a/docs/API.md b/docs/API.md new file mode 100644 index 0000000..e7e26cc --- /dev/null +++ b/docs/API.md @@ -0,0 +1,29 @@ +# TestSwarm API + +As of v1.0.0, TestSwarm provides a Web API. + +It is reachable at `{swarmroot}/api.php` + +## Formats + +* [JSON](https://en.wikipedia.org/wiki/JSON) (**recommended**) +* [JSON-P](https://en.wikipedia.org/wiki/JSONP) (using `callback` parameter) +* Debug (see below) + +### JSON + +See also [http://json.org/](http://json.org/) for a list of decoders in various programming languages. + +### JSON-P + +Passes the JSON to a callback function (named in the `callback` url parameter). This enables use of the TestSwarm API cross-domain in the web browser. + +**NB:** For security reasons and to avoid abuse, the API does not allow logged-in sessions via JSONP. + +### Debug + +The debug format shows an HTML page instead with information about the request and then response is shown in a "pretty" format for human reading using PHPs `var_dump` function. Together with the configuration options `php_error_reporting` and `db_log_queries` in the `[debug]` section of `testswarm.ini`, this is a very wide range debugging interface for API queries. + +## Actions + +A full list of actions can be found in the `./inc/actions` directory of your TestSwarm installation. See the `@actionXXX` comments above the `doAction` methods in each class for more information about how the action should be used and what the required parameters and request methods are. \ No newline at end of file diff --git a/docs/About.md b/docs/About.md new file mode 100644 index 0000000..317f786 --- /dev/null +++ b/docs/About.md @@ -0,0 +1,79 @@ +**TestSwarm** provides distributed continuous integration testing for JavaScript. It was originally created by [John Resig](http://ejohn.org/) as a basic tool to support unit testing of the [jQuery JavaScript library](http://jquery.com). It has since become a [jQuery Foundation](https://jquery.org/) project and is currently maintained by [Timo Tijhof](https://timotijhof.net). + +## Philosophy + + + +The primary goal of TestSwarm is to take the complicated, and time-consuming, process of running JavaScript test suites in multiple browsers and to grossly simplify it. It achieves this goal by providing all the tools necessary for creating a continuous integration workflow for your JavaScript project. + +The ultimate result of TestSwarm are the project pages and job pages.
+ +Project page: + +TestSwarm Project page + +It shows source control commits (going vertically) by browser (going horizontally). 'Green' indicates the runs were are 100% passing, 'Red' indicates a failure, and 'Grey' means the runs are scheduled awaiting run + +For more details on the individual jobs in a project, click the job title in the first column. + +Job page: + +TestSwarm Job page + +This shows all individual runs of the job (going vertically) by browser. To view the run results of a completed run, click the "run results" icon inside the colored cell. + +## Architecture + +### Structure + +From top to bottom, the structure is as follows +* A TestSwarm install has a number of authorized accounts. These are called "users" or "projects" +* Projects have a timeline consisting of jobs. An example of a job would for example be "`jquery:master #5ffa3ea7eea3fd9848fb5833e3306d390672a3e2`" in project "`jquery`". +* Jobs contain two mappings: + * Runs: Every run represents a module that needs to be tested for a job. For example "selectors" or "attributes" are example of runs in a typical jQuery core test job. + * Browsers: Every job has a set of browsers that it needs to validate against. TestSwarm will distribute all runs of a job to those browsers only. + +![TestSwarm diagram](./img/ts-swarm.png) + +The architecture is as follows: + +* Clients join the swarm by connecting to the "run" page on the swarm server. +* The run page is used as the framework to run all tests inside. One user can open multiple clients - and even multiple clients within a single browser. For example you could open 5 tabs in Firefox 3 each with a view of the test runner and would have 5 clients connected. +* The test runner periodically pings the server, asking for the latest "run" for this browser that hasn't been run yet. If there is one, it executes it (inside an iframe) and then submits the results back to the server. If there are no new runs available, the test runner goes back to sleep and tries again later. +* Every job also has a "run max" property. More about that below. + + +### Result correction. + +An important aspect of TestSwarm is its ability to proactively correct bad results coming in from clients. As any web developer knows: Browsers are surprisingly unreliable (inconsistent results, browser bugs, network issues, etc.). Here are a few of the things that TestSwarm does to try and generate reliable results: + +* If a client loses its internet connection or otherwise stops responding its dead results will be automatically cleaned up by the swarm. +* If a client is unable to communicate with the central server it'll repeatedly re-attempt to connect (even going so far as to reload the page in an attempt to clear up any browser-born issues). +* The client has a global timeout to watch for test suites that are uncommunicative. +* The client has the ability to watch for individual test timeouts, allowing for partial results to be submitted back to the server. +* Bad results submitted by clients (e.g. ones with errors, failures, or other timeouts) are automatically re-run in new clients in an attempt to arrive at a passing state (the number of times in which the test is re-run is determined by the submitter of the job). + +All together these strategies help the swarm to be quite resilient to misbehaving browsers, flaky internet connections, or even poorly-written test suites. + +For example if a job has a runmax of 3, and it fails the first time it will distribute it again (preferably to a different client with the same user agent, otherwise the same client will get it again later) until it either passes or hits the maximum of 3. + +## How is TestSwarm Different From... + +### Selenium + +[Selenium](http://seleniumhq.org/) provides a quite-full stack of functionality. It has a test suite, a test driver, automated browser launching, and the ability to distribute test suites to many machines (using their grid functionality). There are a few important ways in which TestSwarm is different: +* TestSwarm is test suite agnostic. It isn't designed for any particular test runner and is capable of supporting any generic JavaScript test suite. +* TestSwarm is much more decentralized. Jobs can be submitted to TestSwarm without clients being connected - and will only be run once clients eventually connect. +* TestSwarm automatically corrects mis-behaving clients and malformed results. +* TestSwarm provides a full continuous integration experience (hooks in to source control and a full browser-by-commit view which is critical for determining the quality of commits). +* TestSwarm doesn't require any browser plugins or extensions to be installed - nor does it require that any software be installed on the client machines. + +For a number of corporations Selenium may already suit your needs (especially if you already have a form of continuous integration set up). + +### JSTestDriver and Other Browser Launchers + +There are many other browser launching tools (such as [Watir](http://wtr.rubyforge.org/)) but all of them suffer from the same problems as above - and frequently with even less support for advanced features like continuous integration. + +### Server-Side Test Running + +A popular alternative to the process of launching browsers and running test suites is that of running tests in headless instances of browsers (or in browser simulations, like in Rhino). All of these suffer from a critical problem: At a fundamental level you are no longer running tests in an actual browser - and the results can no longer be guaranteed to be identical to an actual browser. Unfortunately nothing can truly replace the experience of running actual code in a real browser. \ No newline at end of file diff --git a/docs/Automation.md b/docs/Automation.md new file mode 100644 index 0000000..28f1388 --- /dev/null +++ b/docs/Automation.md @@ -0,0 +1,173 @@ +# Automated Distributed Continuous Integration for JavaScript + +_Just getting started with TestSwarm? Check out the [README](../README.md) instead._ + +This document describes how **TestSwarm** can be used in a large scale testing environment. + +Features: + +* Fully **automated submissions of jobs** to TestSwarm +* The **swarm is automatically populated** with browser clients +* Events can be used to create additional **post-build actions** (such as IRC notifications). + + +Main frameworks: [TestSwarm](https://github.com/jquery/testswarm), [Jenkins](http://jenkins-ci.org/), [QUnit](https://github.com/jquery/qunit) + +Utilities: [node-testswarm](https://github.com/jzaefferer/node-testswarm), [node-browserstack](https://github.com/scottgonzalez/node-browserstack), [testwarm-browserstack](https://github.com/clarkbox/testswarm-browserstack), [grunt](https://github.com/cowboy/grunt) + +**TL;DR: → Check out the [End result](#h-end_result).** + +## Prerequisites + +The following prerequisites are assumed to be installed already (this document does not cover setting that up, these software packages have their own manuals): + +* Install [jQuery TestSwarm](https://github.com/jquery/testswarm) (obviously) + * Create an account for your project in TestSwarm, and take note of the authToken for that account. +* Install [Jenkins](http://jenkins-ci.org/)
with the following plugins: (depending on your application and development workflow you may need different ones) + * [Git Plugin](https://wiki.jenkins-ci.org/display/JENKINS/Git+Plugin) + * [Github Plugin](https://wiki.jenkins-ci.org/display/JENKINS/Github+Plugin) + * [Instant Messaging Plugin](https://wiki.jenkins-ci.org/display/JENKINS/Instant+Messaging+Plugin) + * [IRC Plugin](https://wiki.jenkins-ci.org/display/JENKINS/IRC+Plugin) + * [AnsiColor Plugin](https://wiki.jenkins-ci.org/display/JENKINS/AnsiColor+Plugin) +* Account at BrowserStack (required to use the [BrowserStack API](http://www.browserstack.com/automated-browser-testing-api)) + +Note that none of these are claimed to be the "ultimate" tool. They all have alternatives and competitors. + +For example: + +* Jenkins: It is not required for a working continuous integration environment with TestSwarm. TestSwarm itself is (although basic) a fine continuous integration framework on its own. If you prefer to use TestSwarm (instead of Jenkins) as the central manager of your workflow that is also possible. In that case you will have to find a different way to trigger jobs. For example you could write your own script (e.g. ran from crontab) that checks your incoming source of commits (Self-hosted Git or SVN repo, GitHub, Gerrit, ..) and take care of copying the source at the current revision into a [static directory](#h-static_clone) and submit jobs from there. +* BrowserStack: There are alternatives (check out [this StackOverflow post](http://stackoverflow.com/a/10387902/319266)), but those may or may not have an API that allows automatic starting and terminating of browsers. You can also populate your swarm manually by crowdsourcing it. If you can rely on that, go for it (and that's free, whereas cloudbased solutions like BrowserStack usually aren't). +* QUnit: Developed and used by all jQuery projects and many other projects as well. However, TestSwarm also supports [various other frameworks](https://github.com/jquery/testswarm/tree/master/scripts/addjob#test-suite-runs). + + + +## Static clone + +Due to the asynchronous nature of TestSwarm it is important that tests are ran on a static copy of your source code, fixed on the version that it should be testing. Depending on your project there can be a few ways to do this. Here is an example for jQuery and MediaWiki. + +### jQuery + +[jQuery Core](http://jquery.com/) only contains static files (static, e.g. there are no PHP files that should be ran from a webserver or need database installation). So making a static copy for this is fairly straight forward. All we do is copy the repository directory (except for the `.git` folder) into another directory that is named after the current commit ID, and make is accessible from the web. That location is then submitted to TestSwarm so that all browser clients can run the tests. + +The following is an example of such a script to make a static copy. The script in particular is used by jQuery on http://swarm.jquery.org/ to make static clones of their git repositories after the "build" process is completed (e.g. concatenate all modules and produce the the raw and minified versions): + +```bash +#!/bin/bash + +# Make a static clone the working copy of a Jenkins build +# to a public directory for browser testing. +# Also cleans clones older than 14 days +# +# Parameters: +# $1: Publish target (absolute path, without "$1") +# $2: ID to use for the static clone directory +# +# Usage: +# From within the Jenkins working copy directory. +# +# version 5 (2012-07-31) + +dest=/var/www/builds.jenkins.jquery.com/htdocs + +#echo "Creating a static clone of the current directory." +#echo "Group: $1" +#echo "Copy ID: $2" + +mkdir -p $dest/$1/$2 + +echo "Copying... to " $dest/$1/$2 +rsync -a --exclude=".git*" . $dest/$1/$2 + +echo "Cleaning up outdated copies" +cd $dest/$1 +find . -mindepth 1 -maxdepth 1 -ctime +14 | grep -v master | xargs rm -rf +``` + +### MediaWiki + +If your project is more of a web application that contains PHP files and requires actually being "installed" in order to run the test suite, then this example (based on [MediaWiki](https://www.mediawiki.org/)) may be more helpful. MediaWiki is maintained in a Git repository managed by a Gerrit install at the [Wikimedia Foundation](https://www.wikimedia.org/). All commits are reviewed and tested before being merged into master. More about that: +* [Gerrit Code Review](https://code.google.com/p/gerrit/) +* [Gerrit Trigger](https://wiki.jenkins-ci.org/display/JENKINS/Gerrit+Trigger) (Jenkins plugin) + +The build for MediaWiki goes through several steps. Most steps are not related to making a static copy for TestSwarm, so those steps are simplified here. The summary here is intended as guide of inspiration for your own project +* Fetch from Gerrit ([fetch_gerrit_head.sh](https://gerrit.wikimedia.org/r/gitweb?p=integration/jenkins.git;a=blob;f=bin/fetch_gerrit_head.sh;h=f464d0606a74662b7f2469aa527d6c65964a9cfc;hb=HEAD)) +* Install it + * Using php from the command line to invoke an installation script +* Snapshot ([testswarm-snapshot](https://gerrit.wikimedia.org/r/gitweb?p=integration/jenkins.git;a=blob;f=jobs/_shared/build.xml;h=aee477988fdc5f3fe231e0b35180466ffbe8826d;hb=HEAD#l188)) + * Make a copy (without the `.git` dir) of the working space into a fixed place (e.g. `/srv/integration.mediawiki.org/clones/mw/{gerrit_change_id}/{patch_version}`) + * Copy database (in case of sqlite this is fairly simple, just copying the file. In the case of MySQL it is more complicated. You'll need to rename/move the database to unique name) + * Tweak configuration (update database settings and location-dependent settings) + * Clear location-dependent caches + +## Jenkins + +### Jenkins job configuration: + +* GitHub project: `https://github.com/example/test/` +* Source Code Management: `Git` + * URL of repository: `git://github.com/example/test.git` + * Branch Specifier: `master` +* Build Triggers + * [x] Build when a change is pushed to GitHub +* Build
**This highly depends on your project. This example is for jQuery** + * Execute shell
- installs the dependencies of your project as specified in `package.json` ([jquery example](https://github.com/jquery/jquery/blob/master/package.json))
- executes the main building proces and testing procedure as specified in `Gruntfile.js` ([jquery example](https://github.com/jquery/jquery/blob/master/Gruntfile.js)).
If your only test is TestSwarm, then this "grunt" step is redundant. Examples of things that could be in this "grunt" step are: `jshint` or `jslint`, `csslint`, `htmllint`, "running qunit in phantomjs" etc. +```bash +npm install +grunt +``` + * Execute shell
This is making the static clone, more about that in the previous section. +```bash +/srv/swarm.jquery.org/tools/jenkins-testswarm-static-copy.sh jquery ${GIT_COMMIT} +``` + * Execute shell
This executes the "testswarm" task as defined in `Gruntfile.js` ([jquery example](https://github.com/jquery/jquery/blob/master/Gruntfile.js#L111)). This task uses the `node-testswarm` module to create a request to the TestSwarm installation and creates a new job, using the current Jenkins build as the target. `node-testswarm-config.json` is not related to the `node-testswarm` package itself, but is used by jquery's Gruntfile.js to not hardcode configuration settings in the repository (and for private settings, such as authTokens). +```bash +grunt testswarm:${GIT_COMMIT}:/srv/swarm.jquery.org/tools/node-testswarm-config.json +``` +* Post-build Actions + * [x] IRC Notification (Channels: _optionally override the channels for this build from the main plugin config_) + +### Jenkins plugin configuration + +#### IRC + +* [x] Enable IRC Notification + * Hostname: irc.freenode.net + * Port: 6667 + * Channels: _default list of channels for this project, can be overridden in the job config_ + * Nickname: _nickname of jenkins irc bot_ + +#### GitHub Web Hook + + * [x] Manually manage hook URLs + +## Cron + +Last but not least there are a few things that should happen frequently but have no other place to go. Those are put into crontab. + +### BrowserStack + +Using the [BrowserStack API](https://github.com/browserstack/api) and the [TestSwarm API](https://github.com/jquery/testswarm/wiki/API) it becomes very easy to keep an eye on the swarm and see which browsers are needed, and fill the swarm with the appropriate browsers by instantly creating new virtual machines with the correct OS/browser/browser-version combination and pointing the browser to the TestSwarm run page. This last sentence sounds like a lot of work, but it couldn't be simpler with the BrowserStack API and the [testswarm-browserstack](https://github.com/clarkbox/testswarm-browserstack/) module. + +* Install this module ([readme](https://github.com/clarkbox/testswarm-browserstack#readme)) +* Copy [sample-config.json](https://github.com/clarkbox/testswarm-browserstack/blob/master/sample-config.json) to `config.json` and fill in the variables (swarm location, swarm run page, browserstack credentials). +* Copy [sample-run.sh](https://github.com/clarkbox/testswarm-browserstack/blob/master/sample-run.sh) to `run.sh` +* Add to crontab. + +With just that, every 2 minutes the testswarm-browserstack module will query the TestSwarm API to see which browsers have pending runs, query the BrowserStack API with your credentials and check which browsers are running. Then it will terminate no longer needed browsers, and where not already, start additional browsers to join the swarm. + + +## End result + +The [used programs](#p-techsum) are independent but work really well when put together. This guide is fairly in-depth, but with a little luck you'll be able to set it up in no-time. + +The end result is as follows (note that this workflow is beyond theory, it is the reality for most jQuery projects right now!) + +1. **Commit**
Someone pushes a commit to the repository (or in the case of a system based on GitHub pull-request or Gerrit merge-request, even when someone sends a patch) +2. **Build**
Jenkins will pick up on it and start a build ([example](http://swarm.jquery.org:8080/job/QUnit/119/)) and goes through the steps ([example](http://swarm.jquery.org:8080/job/QUnit/119/console)): + * Execute any necessary building steps and/or run local tests (e.g. lint or qunit) + * Submit job to TestSwarm (with node-testswarm) + * Meanwhile, the cron testswarm-browserstack command will keep querying TestSwarm API and start up missing browsers and let them join the swarm, and stop no longer needed virtual machines. + * Meanwhile, TestSwarm keeps distributing pending runs to the swarm clients, which send results back through the TestSwarm API with AJAX + * node-testswarm detects there are no more pending runs for this job, gets the job runresults and marks the Jenkins build as completed and set the Jenkins build status to passed (or failed) + * Any registered hooks will be ran (notification to irc, a comment on the merge-request with stats, etc.) +3. **Repeat** :) diff --git a/docs/Code.md b/docs/Code.md new file mode 100644 index 0000000..9c13648 --- /dev/null +++ b/docs/Code.md @@ -0,0 +1,53 @@ +# TestSwarm Reference Documentation + +## Classes + +TestSwarm (as of v1.0.0) is programmed in an [object-oriented](https://en.wikipedia.org/wiki/Object-oriented_programming) fashion and based around the principle of "context". A short overview of the main classes follows. + +### BrowserInfo + +Given a user agent string this class parses into an object and extracts information about the client. Such as browser name, version and operating system. It also tries to find the "TestSwarm user agent ID", the key in a browserSet in `localSettings.json`. We use the [ua-parser](https://github.com/tobie/ua-parser) library for parsing. + +### Database + +Although right now the only supported database backend is MySQL, the Database class abstracts most MySQL specific methods so that it is fairly easy to change this later on. It also offers various convenience methods (such as `getOne`, `getRow` and `getRows`), and uses proper context based calls to the underlying php functions (e.g. methods like `mysql_error` and `mysql_insert_id` are always called with the connection object in case multiple connections are active). + +### WebRequest + +Abstraction for accessing request data (`$_GET`, `$_POST`, `$_SESSION` and IP-address). Also provides several convenience methods such as `wasPosted()`, `getBool, `getInt`, `getArray` and `getVal( key, defaultValue )`. + +### TestSwarmContext + +The above and other classes are lazy loaded when they are needed. Both the class files themselves are lazy-loaded using the PHP Autoloader and the initialization of the classes is done on-the-fly when they are first needed and the instance is cached from there for re-use. + +### Action + +The base class for all actions. Actions are the logic independent from the front-end. Pages and the API both use these to get the information in a structured format. + +### Page + +Formats the skin for `index.php` requests and Page-extending classes implement page specific output, gathered from one or more instances of Action classes. + +## Globals + +As of TestSwarm 1.0.0 TestSwarm practically uses zero globals. There are however 2 global variables used in some legacy functions that will be phased out later. + +* `$swarmInstallDir` (string) +This is the absolute path of the directory where the TestSwarm install is located in the file system. + +* `$swarmContext` (TestSwarmContext) +Primary instance of the `TestSwarmContext` class. Should be used as little as possible. In general only within an entry point file (`index.php`, `api.php`). In any other environment the current context should be used from within the enclosing class. + +* `swarmAutoLoadClasses` (array) + +## Settings +See [Settings](./Settings.md) page. + +## Database schema + +* **projects** - Identification and authentication for projects that submit jobs. +* **clients** - Client sessions. Each session stores a freeform name and User-Agent string. +* **jobs** - Jobs created by a project. Each job has a freeform name. When jobs are created, they also describe a list of runs (name and url), and list of browsers in which each test suite should be run. (linked: `projects.id`) +* **runs** - The run names and urls submitted as part of the job. (linked: `jobs.id`) +* **run_useragent** - State for each of the runs for each of the requested browsers. Has a nullable field pointing to the stored result in the `runresults` table. If a run is requested to re-try in case of test failure (or manually reset from the web interface), then fields in this table will be reset to their initial value. (linked: `runs.id`, `runresults.id`) +* **runresults** - Each result from a client that tested a run. There may be multiple results for the same run as jobs may specify a number of retries. Jobs may also be manually reset from the web interface. Old results are kept and permalinks to it will continue to work. (linked: `runs.id`, `clients.id`) \ No newline at end of file diff --git a/docs/History.md b/docs/History.md new file mode 100644 index 0000000..be86ee7 --- /dev/null +++ b/docs/History.md @@ -0,0 +1,44 @@ +# Project history + +TestSwarm was originally created by [John Resig](https://johnresig.com/) as a +basic tool to support unit testing of the [jQuery JavaScript +library](https://jquery.com). It later become a [Mozilla Labs](https://labs.mozilla.com/) project, +and has since moved again to become a [jQuery Foundation](https://jquery.org/history/) project. + +## Visual history + +## v1.1.0 (2023-03) + +| Page | Screenshot +|---|--- +| Clients page | +| Job page | + +## v1.0.0-alpha (2013-06) + +| Page | Screenshot +|---|--- +| Home page | +| Clients page | +| Result page |
+| Job page | +| Project page | +| Projects page| +| Client page | + +## v1.0.0-alpha (2012-11) + +| Page | Screenshot +|---|--- +| Home page | +| Project page | +| Job page | + +## v0.1.0 (2009) + +* [Video: TestSwarm Walkthrough](https://vimeo.com/6281121), John Resig, 26 August 2009. +* [TestSwarm Alpha Open!](https://johnresig.com/blog/test-swarm-alpha-open/), John Resig, 26 August 2009. + +| Page | Screenshot +|---|--- +| Home page | diff --git a/docs/Settings.md b/docs/Settings.md new file mode 100644 index 0000000..9282a36 --- /dev/null +++ b/docs/Settings.md @@ -0,0 +1,67 @@ +## Settings + +Refer to [`./config/defaultSettings.json`](../config/defaultSettings.json) for the default values of these settings. + + +## General settings +***NOTE***: Settings under section "general" are publicly retrievable through `api.php?action=info`. + +* `general.timezone` (string): Timezone that things should be displayed in. See also http://php.net/date_default_timezone_set. + + +## Database settings +A MySQL connection will be established to specified host. The database has to be set up before the application can be used. Refer to the README for more information. + +* `database.host`: Hostname of MySQL server to connect to. +* `database.database`: Name of the database to use. +* `database.username`: Username to connect with to the above host must have permission on the above database to: `SELECT`, `INSERT`, `UPDATE` and `DELETE`. +* `database.password`: Password for the above username. + + +## Customizable interface messages +The `_html` or `_text` suffix indicates whether it is outputted as-is or html-escaped. + +* `customMsg.homeIntro_html`: Message outputted on the HomePage, wrapped in `

..

`. `$1` is replaced with an html-escaped value of `web.title`. + + +## Web Settings +***NOTE***: Settings under section "web" are publicly retrievable through `api.php?action=info`. + +Settings for the web front-end (index.php) + +* `web.server`: The server address including protocol and (if needed) the port. No trailing slash. When set to `null` (default), `init.php` will do a basic attempt to guess the right value. If urls are broken in your install, make sure to set this correctly. +* `web.contextpath`: The context path is the prefix (starting at the domain name root) used to access resources from the browser. Make sure to also update `.htaccess` with the correct RewriteBase. Should be the absolute path from the root (e.g. `"/"`, or `"/testswarm/"`). +* `web.title`: Name for this TestSwarm instance. Will be used in the header of the site. +* `web.ajaxUpdateInterval`: Number of seconds to wait between AJAX refreshes in web interface (such as the run results table on the Job page). + + +## Client settings +***NOTE***: Settings under section "client" are publicly retrievable through `api.php?action=info`. + +***NOTE***: These are publicly exported on the Run page. When clients request new tests from API they also refresh the client settings. So updating these values affects all connected clients as well. + +* `client.cooldownSleep` (number): After a run has been completed, wait this number of seconds before requesting the next run from the server. +* `client.nonewrunsSleep` (number): If server request for next run gave no results "No new tests to run", wait this number of seconds before trying again. +* `client.runTimeout` (number): Number of seconds a run may take to run. After this time the run-iframe will be cancelled and a timeout-report is submitted to `action=saverun instead. +* `client.saveReqTimeout` (number): Number of seconds the AJAX request to action=saverun on RunPage may take +; until it is aborted. +* `client.saveRetryMax` (number): If the AJAX request to action=saverun on RunPage fails entirely (e.g. times out, 404 Not Found, 500 Internal Server Error, ..) it will retry up to this number of times before reaching out to the last resort (restarting the browser window). When in doubt, keep a higher rather than a lower value. Because it's better to have a client in a still-alive page retrying a few times until the server is back on, then refreshing too soon when the RunPage doesn't even render (which effectively permanently disconnects the client). +* `client.saveRetrySleep` (number): Number of seconds to wait between retries for `action=saverun`. +* `client.requireRunToken` (boolean): Whether joining the swarm to run tests requires a token or not. By default anyone can join the swarm without needing a token or a registered account. When enabling this, run the refreshRunToken.php script to generate a token that will then have to be passed as "`run_token`" for GetrunAction, SaverunAction and on RunPage. +* `client.refreshControl` (number): Increasing this number will force all connected clients to refresh the window. Use this sparingly as a client will not be able to automatically reconnect if the refresh fails for whatever reason. When using AJAX to request new runs, there is a fallback, but for a complete refresh there is no catch. + + +## Storage +* `storage.cacheDir`: Which directory to use for caching. Should be readable/writeable and not executable from the web. Either outside the web root or protected by a restriction from `.htaccess`. As convenience "`$1`" will be replaced with the value of `$swarmInstallDir`. + + +## BrowserSets +***NOTE***: Define your own browserSets in the `localSettings` file. + +[Example](../config/sample-localSettings.json#L11) + + +## Debugging +* `debug.showExceptionDetails` (boolean): Wether to show exceptions in HTML output. By default Uncaught exceptions will be handled by PHP and cause a PHP Fatal error. Depending on your server settings these usually result in the request being aborted and the browser is given a HTTP 500 Internal Server Error (blank page unless [`display_errors`](http://www.php.net/manual/en/errorfunc.configuration.php#ini.display-errors) is on). TestSwarm catches these on a high level, and setting this to `true` will output the exception details. +* `debug.phpErrorReporting` (boolean): If enabled, php errors of all levels will be shown. Note that will allow PHP to output E_NOTICE and E_WARN errors in unexpected places and cause invalid HTML or distorted pages. +* `debug.dbLogQueries` (boolean): If enabled, the Database class will log all queries and the Page class will output them at the bottom of the HTML. In order to use this for [Api](API) responses, use `api?format=debug` which uses a Page as container. Be careful not to use this on a production site as this will show raw queries in HTML and could potentially contain information that users should not be able to see. \ No newline at end of file diff --git a/docs/TestResult-Specification.md b/docs/TestResult-Specification.md new file mode 100644 index 0000000..26264fa --- /dev/null +++ b/docs/TestResult-Specification.md @@ -0,0 +1,254 @@ +# TestResult Format Specification Version 1 + +## Editor's Draft 11 July 2012 +
+
This version
+
https://github.com/jquery/testswarm/blob/main/docs/TestResult-Specification.md
+
Editors
+
Timo Tijhof
+
+ +## 1. Introduction + +This document specified the structure that a TestResult Report (hereafter referred to as "report") must follow. It is intended to be lightweight and usable for summarizing test results as generated by any compatible application for performing unit tests ("frameworks"). There are very few requirements of the report format, so it is expected to be compatible with most of the popular unit test frameworks in the world today. + +Those that create or otherwise use reports (hereafter referred to as "user") can transport these reports between applications in a format of their choosing (as long as it supports nested key/value pairs, where a key can be any string, and a value a number, boolean, string, array or object). The recommended data-interchange format is JSON (see also [www.json.org](http://www.json.org/) / [RFC 4627](https://www.ietf.org/rfc/rfc4627.txt)). This document will use JSON in examples. Inside an application flow the user should use a native language structure if possible (e.g. objects in JavaScript, associative arrays in PHP, etc.). + +The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in [RFC 2119](http://www.ietf.org/rfc/rfc2119.txt). + +## 2. Structures +The report must have exactly one top-most group of key/value pairs (hereafter referred to as "objects" having "properties") that follows the specification for the "Root" structure, as specified in section 2.1 below. + +All of these objects specified below have one common requirement. They must have all of the described properties. Additional properties not specified below are not permitted. And, with the exception of those marked as "optional", none of the properties specified below may be permitted. + +### 2.1 Root +Both the `groups` and `assertions` properties are optional. This allows for usage for simple linear test suites, as as well test suites that have a deeply nested hierarchy. Both may be present simultaneously, which indicates that some assertions are not contained in a group. Though both are optional, the root must contain at least one of `groups` or `assertions`. + +#### 2.1.1 name +* Required +* Content: Name String + +#### 2.1.2 summary +* Required +* Content: Summary Object + +#### 2.1.3 time +* Optional +* Content: Number + +The total run-time of all assertions in this test suite (in milliseconds). This sum should match the recursively calculated sum of all assertion times. However it does not have to match that sum. The root time may include execution time between assertions (like a setup or teardown for a group or the test suite as a whole) if a user chooses to do so. + +#### 2.1.4 groups +* Optional +* Content: Array containing Group objects + + +#### 2.1.5 assertions +* Optional +* Content: Array containing Assertion objects + +### 2.2 Summary +The values of a summary object should reflect the sum of all containing groups and/or assertions (recursively). + +#### 2.2.1 total +* Required +* Content: Number + +Total number of assertions in this group. Should be recursive to include nested assertions if the group contains other groups. + +#### 2.2.2 failed +* Required +* Content: Number + +Number of assertions that are to be considered "failing the test suite". The user's framework may have different types of unsuccessful results. It is up to the user to decide whether or not to include a non-passing result in the this total. + +For example, in PHPUnit, there are annotations to indicate known "Broken" or "Incomplete" tests, which are skipped during execution or other wise excluded from the "failure" count. + +### 2.3 Group + +#### 2.3.1 name +* Required +* Content: Name String + +#### 2.3.2 summary +* Required +* Content: Summary Object + +#### 2.3.3 time +* Optional +* Content: Number + +Total run-time of all containing assertions (in milliseconds). May or may not include execution time between, before and/or after assertions within the boundaries of the group. + +#### 2.3.4 groups +* Optional +* Content: Array containing Group objects + +#### 2.3.5 assertions +* Optional +* Content: Array containing Assertion objects + +### 2.4 Assertion + +#### 2.4.1 name +* Required +* Content: Name String + +#### 2.4.2 source +* Optional +* Content: Source Object + +#### 2.4.3 time +* Optional +* Content: Number + +Run-time in milliseconds. + +#### 2.4.4 status +* Required +* Content: String + +Summarizes the outcome of the assertion. Must be one of: +* `pass`: The assertion passed as expected +* `fail`: The assertion failed to meet the expected outcome + +No non-standard values allowed here. + +#### 2.4. result +* Optional +* Content: Result Object + +### 2.5 Source + + + +### 2.6 Result + + + +### 2.7 Name + +Free form text. Title representing the object the property belongs to. Must not be used for unique identification. May contain some kind of identification for human readers to identify a certain version of build, but does not have to. + +For example, a test suite may be named like: +* "ACME Project Test Suite" +* "ACME 2.0-beta2 Test Suite" +* "ACME 2.0 Test Suite (04708c0)" + +And similarly there may be multiple assertions with the same name. For example there could be a group of tests that tests an object in different ways after performing a certain operation. Then another test group could run those same group of assertion when doing a different operation: +* Group: FooBar applies quuxification to x + * Assert: fb.quuxified matches expected state + * Assert: ... +* Group: FooBaz does not quuxify y + * Assert: fb.quuxified matches expected state + * Assert: ... + + + + + \ No newline at end of file diff --git a/docs/img/2009-home.png b/docs/img/2009-home.png new file mode 100644 index 0000000..47bdb48 Binary files /dev/null and b/docs/img/2009-home.png differ diff --git a/docs/img/2012-home.png b/docs/img/2012-home.png new file mode 100644 index 0000000..2f4bb01 Binary files /dev/null and b/docs/img/2012-home.png differ diff --git a/docs/img/2012-job.png b/docs/img/2012-job.png new file mode 100644 index 0000000..fe26c87 Binary files /dev/null and b/docs/img/2012-job.png differ diff --git a/docs/img/2012-project.png b/docs/img/2012-project.png new file mode 100644 index 0000000..0cddcb3 Binary files /dev/null and b/docs/img/2012-project.png differ diff --git a/docs/img/2013-client.png b/docs/img/2013-client.png new file mode 100644 index 0000000..9fb4a4b Binary files /dev/null and b/docs/img/2013-client.png differ diff --git a/docs/img/2013-clients-names.png b/docs/img/2013-clients-names.png new file mode 100644 index 0000000..13e7e7c Binary files /dev/null and b/docs/img/2013-clients-names.png differ diff --git a/docs/img/2013-clients.png b/docs/img/2013-clients.png new file mode 100644 index 0000000..2465798 Binary files /dev/null and b/docs/img/2013-clients.png differ diff --git a/docs/img/2013-home-open.png b/docs/img/2013-home-open.png new file mode 100644 index 0000000..c7a55b5 Binary files /dev/null and b/docs/img/2013-home-open.png differ diff --git a/docs/img/2013-home-restricted.png b/docs/img/2013-home-restricted.png new file mode 100644 index 0000000..b9c7a9f Binary files /dev/null and b/docs/img/2013-home-restricted.png differ diff --git a/docs/img/2013-job.png b/docs/img/2013-job.png new file mode 100644 index 0000000..41f9d5f Binary files /dev/null and b/docs/img/2013-job.png differ diff --git a/docs/img/2013-project.png b/docs/img/2013-project.png new file mode 100644 index 0000000..dae0960 Binary files /dev/null and b/docs/img/2013-project.png differ diff --git a/docs/img/2013-projects.png b/docs/img/2013-projects.png new file mode 100644 index 0000000..8e3d6df Binary files /dev/null and b/docs/img/2013-projects.png differ diff --git a/docs/img/2013-runresult.png b/docs/img/2013-runresult.png new file mode 100644 index 0000000..02b6aeb Binary files /dev/null and b/docs/img/2013-runresult.png differ diff --git a/docs/img/2013-runresult2.png b/docs/img/2013-runresult2.png new file mode 100644 index 0000000..2837d25 Binary files /dev/null and b/docs/img/2013-runresult2.png differ diff --git a/docs/img/2023-clients-finished.png b/docs/img/2023-clients-finished.png new file mode 100644 index 0000000..ebadf76 Binary files /dev/null and b/docs/img/2023-clients-finished.png differ diff --git a/docs/img/2023-clients-running.png b/docs/img/2023-clients-running.png new file mode 100644 index 0000000..0b09d86 Binary files /dev/null and b/docs/img/2023-clients-running.png differ diff --git a/docs/img/2023-job.png b/docs/img/2023-job.png new file mode 100644 index 0000000..36b0071 Binary files /dev/null and b/docs/img/2023-job.png differ diff --git a/docs/img/ts-swarm.png b/docs/img/ts-swarm.png new file mode 100644 index 0000000..79aa59b Binary files /dev/null and b/docs/img/ts-swarm.png differ