Common Reporter Interface - Working Draft

Participate: GitHub qunitjs/js-reporters (new issue, open issues)
Chat room on Gitter
Last updated: 20 Februrary 2021
Abstract: This specification defines JavaScript APIs for reporting progress and results of executing software tests.
Goal: This specification is meant to be implemented by testing frameworks, reporters, and middleware adapters.

1. Terminology
2. Events
3. Event data
4. Producer API
- 4.1. producer.on(eventName, callback)
5. Reporter API
- 5.1. reporter.init(producer)
6. Acknowledgments

1. Terminology

Testing framework: A testing framework is a program that helps define, organize, load, or execute software tests through assertions. (Learn more)
Adapters: A program that implements the Producer API on behalf of a testing framework, for example to support testing frameworks that don’t yet implement the CRI standard, or to support reporting events from a remotely-executed test run.
Producer: Any JavaScript program that implements the Producer API of the CRI standard and emit its events, typically this is a testing framework, or an adapter for one.
Assertion: An assertion is a logical preposition required to evaluate to true. Assertions must be part of a "Test". (Learn more)
Passing assertion: An assertion that has evaluated to boolean true.
Failed assertion: An assertion that has evaluated to boolean false.
Test: A test is a named group containing zero or more assertions.

It is recommended that all tests contain assertions, but this is not required. For example, a testing framework that only records failed assertions (such as a testing framework that is decoupled from an assertion library and uses exceptions to discover failures), might not distinguish between a test with passing assertions and a test with no assertions. If a testing framework is generally aware of assertions and if it considers absence of those an error, then it should ensure the test or test is marked as failing. For example, by implicitly producing a failed assertion.

In QUnit, a test may be defined by calling QUnit.test().
In Mocha and Jasmine BDD, a test is known as a "spec", defined by calling it().
(Learn more)
Skipped test: A Test that was not actually run. Testing frameworks may have ways of selecting, partially loading, filtering, or otherwise skipping tests. These implementation choices may mean that some tests are not considered part of the Run, and thus entirely left out of the information exposed to reporters. Presence of one skipped test does not imply that all skipped tests will be reported in this way.

See also the SKIP directive of the TAP specification.
Todo test: A Test that is expected to have one or more failing assertions.

See also the TODO directive of the TAP specification.
Suite: A suite is a named group representing zero or more tests, and zero or more other suites. A suite that is part of another suite may also be called a "child suite". A suite that holds one or more child suites may also be called an "ancestor suite".
(Learn more)

In QUnit, a suite is known as a "module", defined by calling QUnit.module().
In Mocha and Jasmine BDD, a suite is defined by calling describe().
In JUnit and other xUnit-derivatives, tests are first grouped in a TestCase which are then further grouped under a <testsuite>. In the CRI standard, both of these are considered a suite.
Run: A run is a single top-level group representing all tests and suites that a producer is planning to report events about.
Reporter: A JavaScript program that consumes information from a Run. For example, to render an HTML graphical user interface, to write command-line output in the TAP format, write results to a JUnit XML artifact file, or serialize the information and transfer it over a socket to another server or process.

💡	The use of "Suite" and "Test" as the main two data structues was decided in issue #12, and later revised in issue #126.

2. Events

These are the events that a [producer] should emit from its Producer API, for consumption by a [reporter].

💡

These events were selected as:

common across known testing frameworks (gathered in issue #1).
valid JavaScript identifiers, allowing use as variable name and as object literal key without quotes.
not overlapping with existing events in known testing frameworks, for easy adoption within existing APIs.

It is recommended, though not required, that events about tests are emitted in source order, based on how the tests are defined by a developer in a test file. This means results of tests defined is higher up in a test file should be emitted earlier than those defined lower down in the file.

Note that execution order may be different from reporting order. If a testing framework uses concurrency or random seeding for its execution, we recommend that events are still consistently emitted in the source order.

💡	Read issue #62 for the discussion about reporting order.

2.2. `runStart` event

The runStart event indicates the beginning of a Run. It must be emitted exactly once, and before any suiteStart event or testStart event.

Callback parameters:

RunStart runStart: The plan for the run.

producer.on('runStart', (runStart) => { … });

2.3. `runEnd` event

The runEnd event indicates the end of a Run. It must be emitted exactly once, after the last of any suiteEnd event or testEnd event.

Callback parameters:

RunEnd runEnd: Summary of test results from the completed run.

producer.on('runEnd', (runEnd) => { … });

2.4. `suiteStart` event

The suiteStart event indicates the beginning of a Suite. It must eventually be followed by a corresponding suiteEnd event.

Callback parameters:

SuiteStart suiteStart: Basic information about a suite.

producer.on('suiteStart', (suiteStart) => { … });

2.5. `suiteEnd` event

The suiteEnd event indicates the end of a Suite. It must be emitted after its corresponding suiteStart event.

Callback parameters:

SuiteEnd suiteEnd: Basic information about a completed suite.

producer.on('suiteEnd', (suiteEnd) => { … });

2.6. `testStart` event

The testStart event indicates the beginning of a Test. It must eventually be followed by a corresponding testEnd event. A producer may emit several testStart events before any corresponding testEnd event, for example when there are child tests, or tests that run concurrently.

Callback parameters:

TestStart testStart: Basic information about a test.

producer.on('testStart', (testStart) => { … });

💡	If a producer has no real-time information about test execution, it may simply emit `testStart` back-to-back with `testEnd`.

2.7. `testEnd` event

The testEnd event indicates the end of a Test. It must be emitted after its corresponding testStart event.

Callback parameters:

TestEnd testEnd: Result of a completed test.

producer.on('testEnd', (testEnd) => { … });

3. Event data

The following data structures must be implemented as objects that have the specified fields as own properties. The objects are not required to be an instance of any specific class. They may be null-inherited objects, plain objects, or an instance of any public or private class.

3.1. SuiteStart

SuiteStart object:

string name: Name of the suite.
Array<string> fullName: List of one or more strings, containing (in order) the names of any grandancestor suites, the name of the suite.

3.2. SuiteEnd

SuiteEnd object:

string name: Name of the suite.
Array<string> fullName: List of one or more strings, containing (in order) the names of any grandancestor suites, the name of the suite.
string status: Aggregate result of all tests, one of:
- failed if at least one test has failed.
- passed, if there were no failed tests, which means there either were no tests, or tests only had passed, skipped, or todo statuses.
number|null runtime: Optional duration of the suite in milliseconds.

3.3. RunStart

The plan for the Run.

RunStart object:

string|null name: Name of the overall run, or null if the producer is unaware of a name.
Object testCounts: Aggregate counts about tests.
- number|null total: Total number of tests the producer is expecting to emit events for, e.g. if there would be no unexpected failures. It may be null if the total is not known ahead of time.

3.4. RunEnd

Summary of test results from the completed Run.

RunEnd object:

string|null name: Name of the overall run, or null if the producer is unaware of a name.
string status: Aggregate result of all tests, one of:
- failed if at least one test has failed.
- passed, if there were no failed tests, which means there either were no tests, or tests only had passed, skipped, or todo statuses.
Object testCounts: Aggregate counts about tests.
- number passed: Number of passed tests.
- number failed: Number of failed tests.
- number skipped: Number of skipped tests.
- number todo: Number of todo tests.
- number total: Total number of tests, the sum of the above properties must equal this one.
number|null runtime: Optional duration of the run in milliseconds. This may be the sum of the runtime of each test, but may also be higher or lower. For example, it could be higher if the producer includes time spent outside specific tests, or lower if tests run concurrently and the reporter measures observed wall time rather than a sum.

3.5. TestStart

Basic information about a Test.

TestStart object:

string name: Name of the test.
string|null suiteName: Name of the suite the test belongs to, or null if it has no suite.
Array<string> fullName: List of one or more strings, containing (in order) the names of any grandancestor suites, the name of the suite, and the name of the test itself.

3.6. TestEnd

Result of a completed Test. This is a superset of TestStart.

TestEnd object:

string name: Name of the test.
string|null suiteName: Name of the suite the test belongs to, or null if it has no suite.
Array<string> fullName: List of one or more strings, containing (in order) the names of any ancestor suites, the name of the suite, and the name of the test itself.
string status: Result of the test, one of:
- passed, if all assertions have passed, or if no assertions were recorded.
- failed, if at least one assertion has failed or if the test is todo and its assertions unexpectedly all passed.
- skipped, if the test was intentionally not run.
- todo, if the test is todo and indeed has at least one failing assertion still.
number|null runtime: Optional duration of the run in milliseconds.
Array<Assertion> errors: List of failed Assertion objects. It should contain at least one item for failed tests, and must be empty for other tests.
Array<Assertion> assertions: List of failed and any passed Assertion objects. For a skipped test, this must be empty.

3.7. Assertion

The Assertion object contains information about a single assertion.

Assertion object:

boolean passed: Set to true for a passed assertion, false for a failed assertion.
Mixed actual: The actual value passed to the assertion, should be similar to expected for passed assertions.
Mixed expected: The expected value passed to the assertion, should be similar to actual for passed assertions.
string message: Name of the assertion, or description of what the assertion checked for.
string|null stack: Optional stack trace. For a "passed" assertion, the property must be set to null.

Producers may set additional (non-standard) properties on Assertion objects.

💡	The properties of the Assertion object was decided in issue #79, and later revised by issue #105.

4. Producer API

The object on which the Producer API is implemented does not need to be exclusive or otherwise limited to the Producer API. Producers are encouraged to implement the API as transparently as possible.

💡

For example, a testing framework that provides its main interface through a singleton or global object, could implement the Producer API within that interface. In QUnit, producer.on() is implemented as QUnit.on().

If the testing framework works through instantiation or through an "environment" instance (such as Jasmine), the Producer API could be implemented by such object instead.

4.1. producer.on(eventName, callback)

Register a callback to be called whenever the specified event is emitted, as described under Events. May be called multiple times, to register multiple callbacks for a given event.

Parameters:

string eventName: Name of any CRI standard event.
Function callback: A callback function.

Return:

Mixed: May be undefined, or any other value.

💡

The on() method does not need to be exclusive to CRI standard events. The same event emitter may support other events.

In Node.js, the built-in events module provides an EventEmitter that could serve as the basis for a Producer API implementation. For example:

const EventEmitter = require('events');
const producer = new EventEmitter();

// producer.emit('runStart', { … });
// producer.emit('runEnd', { … });

module.exports = producer;

5. Reporter API

The Reporter API can be implemented in as a plain object, a class with static a method, or as exported function.

5.1. reporter.init(producer)

Attach the reporter to the Producer.

Parameters:

Producer producer: The main interface of the testing framework.

Return:

undefined: Void.

Example: Class-based reporter

Example: Functional reporter

class MyReporter {
  constructor (producer) {
    // producer.on(…, …);
  }

  static init (producer) {
    new MyReporter(producer);
  }
}

// CommonJS:
module.exports = MyReporter;

// ES Module:
export default MyReporter;

function init (producer) {
  // producer.on(…, …);
}

// CommonJS:
module.exports = { init: init };

// ES Module:
export { init };

6. Acknowledgments

The editors would like to thank the following people for their contributions to the project: James M. Greene, Jörn Zaefferer, Franziska Carstens, Jiahao Guo, Florentin Simion, Nikhil Shagrithaya, Trent Willis, Kevin Partington, Martin Olsson, jeberger, Timo Tijhof, and Robert Jackson.

This standard is written by Jörn Zaefferer, Timo Tijhof, Franziska Carstens, and Florentin Simion.

Copyright JS Reporters. This text is licensed under the MIT license.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

cri-draft.adoc

cri-draft.adoc

Common Reporter Interface - Working Draft

1. Terminology

2. Events

2.1. Reporting order

2.2. `runStart` event

2.3. `runEnd` event

2.4. `suiteStart` event

2.5. `suiteEnd` event

2.6. `testStart` event

2.7. `testEnd` event

3. Event data

3.1. SuiteStart

3.2. SuiteEnd

3.3. RunStart

3.4. RunEnd

3.5. TestStart

3.6. TestEnd

3.7. Assertion

4. Producer API

4.1. producer.on(eventName, callback)

5. Reporter API

5.1. reporter.init(producer)

6. Acknowledgments

Files

cri-draft.adoc

Latest commit

History

cri-draft.adoc

File metadata and controls

Common Reporter Interface - Working Draft