Implementation of mutation levels

docs/configuration.md

@@ -438,12 +438,13 @@ _Note:_ It is **not** possible to combine mutation range with a [globbing expres
 
 Default: `{}`<br />
 Command line: _none_<br />
-Config file: `"mutator": { "plugins": ["classProperties"], "excludedMutations": ["StringLiteral"] }`
+Config file: `"mutator": { "plugins": ["classProperties"], "includedMutations": ["MutationSpecification"], "excludedMutations": ["MutationSpecification"] }`
 
 - `plugins`: allows you to override the default [babel plugins](https://babeljs.io/docs/en/plugins) to use for JavaScript files.
   By default, Stryker uses [a default list of babel plugins to parse your JS file](https://github.com/stryker-mutator/stryker-js/blob/master/packages/instrumenter/src/parsers/js-parser.ts#L8-L32). It also loads any plugins or presets you might have configured yourself with `.babelrc` or `babel.config.js` files.
   In the rare situation where the plugins Stryker loads conflict with your own local plugins (for example, when using the decorators and decorators-legacy plugins together), you can override the `plugins` here to `[]`.
-- `excludedMutations`: allow you to specify a [list of mutator names](https://stryker-mutator.io/docs/mutation-testing-elements/supported-mutators/#supported-mutators) to be excluded (`ignored`) from the test run. See [Disable mutants](./disable-mutants.md) for more options of how to disable specific mutants.
+- `includedMutations`: allow you to specify a [list of mutator names](https://stryker-mutator.io/docs/mutation-testing-elements/supported-mutators/#supported-mutators), mutation operators, or mutation level to be included in the test run. This will exclude anything not specified in this list.
+- `excludedMutations`: allow you to specify a [list of mutator names](https://stryker-mutator.io/docs/mutation-testing-elements/supported-mutators/#supported-mutators) to be excluded (`ignored`) from the test run. See [Disable mutants](./disable-mutants.md) for more options of how to disable specific mutants. In case `includedMutations` is also specified, this will exclude mutation operators from that list.
 
 _Note: prior to Stryker version 4, the mutator also needed a `name` (or be defined as `string`). This is removed in version 4. Stryker now supports mutating of JavaScript and friend files out of the box, without the need for a mutator plugin._
 

docs/mutation-levels.md

@@ -0,0 +1,75 @@
+---
+title: Mutation Levels
+custom_edit_url: https://github.com/stryker-mutator/stryker-js/edit/master/docs/mutation-levels.md
+---
+
+This page describes the concept of mutation levels and how to use them in your configuration.
+
+## Terminology
+The smallest unit in mutation testing is the **mutation operator**. This is a single type of mutation, like `AdditionOperatorNegation`, which changes a `+`-operator into a `-`-operator.
+
+Every mutation operator belongs to a **mutator**, also referred to as a **mutator group**. This is a set of mutation operators that can be applied to the same node type. 
+For example, the `AdditionOperatorNegation` mutation operator belongs to the `ArithmeticOperator` mutator group.
+
+Finally, a **mutation level** is an artificial grouping of mutation operators with the purpose of striking a balance between performance and efficacy of a mutation run. 
+Such a level is not necessarily in line with the previously mentioned mutator groups, but designed to work right away.
+Currently, mutation levels are named from `Level1` to `Level3`, where `Level1` has the best performance and `Level3` has the best efficacy.
+
+## Specifying included/excluded mutators
+By default, all of Stryker's mutators will be run on your project, which gives the maximum efficacy but also takes the most resources to run. 
+If you want to enable mutation levels, you can choose a level from 1 to 3, like this: ``includedMutators: ['@Level1']``.
+For most users, this should suffice without further tweaks as these mutation levels are designed based upon a representable sample of JS and TS projects.
+
+In case you want a more advanced and customized configuration, you can tweak your selected mutation level by adding or removing mutation operators and/or mutator groups.
+A **mutation operator** can be specified with its literal name. **Mutator groups** and **mutation levels** are specified with the `@` prefix, for example `@ArithmeticOperator` or `@Level1`.
+For example, if you want to tweak level 2 by removing all `ArithmeticOperator`'s mutation operators except for the `AdditionOperatorNegation`, you would write this: 
+```
+   includedMutators: ['@Level1', 'AdditionOperatorNegation'],
+   excludedMutators: ['@ArithmeticOperator']
+```
+When making these customization tweaks, it is recommended to test the efficacy and performance of these tweaks against the base level to see whether they make a significant enough effect.
+
+## How to reason about modifying a mutation run
+
+A mutation run can be modified such that either the execution time is faster, or the number of covered tests is higher. Unfortunately, these two properties cannot occur at the same time, since they are inversely proportional: as the performance of a run increases, fewer test cases will be executed, which will result in lower coverage.
+
+### Using predefined levels
+
+Stryker pre-defines a few mutation levels such that they provide an attractive range of efficiency-performance tradeoffs. A multitude of Javascript/TypeScript projects was used for designing these levels, and chances are that they will be suitable for your project as well. For this reason, restricting a mutation run by using a predefined level should be the first option that you should consider when you desire to gain additional performance. 
+
+### Customized configuration
+
+However, it could be that the predefined levels do not provide suitable results for your project, and you need to further customize the configuration. Although the previous sections specify the syntax for including/excluding mutators, they do not provide the intuition on how to exactly pick the most suitable choices. 
+
+To find these best choices, we will use an external tool called [Callisto](https://github.com/stryker-mutator/callisto), which is used to quantify the resolution and the performance impact of mutation operators. This is a CLI tool that takes as input the JSON mutation report generated by Stryker and outputs a CSV file with several statistics (quality, performance impact, mutant count, etc) for each mutation operator. Callisto can be used in the following manner:
+
+
+1. Generate JSON Report with Stryker
+
+    Currently, only StrykerJS generates a JSON report which is directly compatible with Callisto. To ensure that Stryker has the correct configuration for generating the JSON file, you need to make sure that the following options are selected in the configuration:
+
+    ```json
+    {
+        ...
+        "disableBail": true,
+        "coverageAnalysis": "perTest",
+        "reporters": [..., "json", ...]
+        ...
+    }
+    ```
+
+2. (Optionally) Correct the JSON report
+
+    Some testing frameworks that StrykerJS uses might result in occasionally small mistakes, which prevents Callisto from deducing a mutation operator name. If there are any such mistakes, Callisto will report any mutants for which it cannot determine a mutation operator name through the terminal. The JSON report needs to be manually corrected with the reported errors before moving on to the next step.
+
+3. Run Callisto on the JSON file to determine statistics for each mutation operator. For details on how to do this, please refer to the documentation of the Callisto tool.
+
+4. Inspect the results
+
+Using the resolution and performance impact metrics, you can make decisions about which mutator/groups to include or exclude from your project. 
+
+For example, if you would like to shorten the time a mutation run takes, take a look at the performance metric and add to the `excludedMutations` list the mutators/groups with the highest performance impact. 
+
+Similarly, you might want to execute more test cases. Then, you should look at the quality metric and add the mutators with the highest values to the `includedMutations` list. However, this metric needs to be inspected together with its mutation count; consider the scenario where an operator has a high-quality score but a small number of generated mutants. Then, the measurement is not very reliable. 
+
+Note that if you are using one of the predefined levels, some mutators might be already included or excluded.    

e2e/test/ignore-project/stryker.conf.json

@@ -4,7 +4,8 @@
   "concurrency": 2,
   "coverageAnalysis": "perTest",
   "mutator": {
-    "excludedMutations": ["ArithmeticOperator", "BlockStatement"]
+    "includedMutations": ["@StringLiteral", "@ConditionalExpression", "@EqualityOperator", "@LogicalOperator", "@BooleanLiteral"],
+    "excludedMutations": ["@ArithmeticOperator", "BlockStatementRemoval"]
   },
   "reporters": [
     "clear-text",

e2e/test/ignore-project/verify/verify.js

@@ -30,14 +30,37 @@ describe('After running stryker on jest-react project', () => {
     });
   });
 
-  it('should report mutants that result from excluded mutators with the correct ignore reason', async () => {
+  it('should report mutants that are excluded from the excludedMutation list with the correct ignore reason', async () => {
     const report = await readMutationTestingJsonResult();
     const circleResult = report.files['src/Circle.js'];
     const mutantsAtLine3 = circleResult.mutants.filter(({ location }) => location.start.line === 3);
     expect(mutantsAtLine3).lengthOf(2);
     mutantsAtLine3.forEach((mutant) => {
       expect(mutant.status).eq('Ignored');
-      expect(mutant.statusReason).eq('Ignored because of excluded mutation "ArithmeticOperator"');
+      expect(mutant.statusReason).eq('Ignored because the operator "MultiplicationOperatorNegation" is excluded from the mutation run');
+    });
+  });
+
+  it('should report mutants that are excluded because they were not in the includedMutations list', async () => {
+    const report = await readMutationTestingJsonResult();
+    const addResult = report.files['src/Add.js'];
+    const mutantsAtLine7 = addResult.mutants.filter(({ location }) => location.start.line === 7);
+    const updateOperatorMutants = mutantsAtLine7.filter(({ mutatorName }) => mutatorName === 'UpdateOperator');
+
+    const mutantsAtLine14 = addResult.mutants.filter(({ location }) => location.start.line === 14);
+    const unaryOperatorMutants = mutantsAtLine14.filter(({ mutatorName }) => mutatorName === 'UnaryOperator');
+
+    expect(updateOperatorMutants).lengthOf(1);
+    expect(unaryOperatorMutants).lengthOf(1);
+
+    updateOperatorMutants.forEach((updateMutant) => {
+      expect(updateMutant.status).eq('Ignored');
+      expect(updateMutant.statusReason).eq('Ignored because the operator "PostfixIncrementOperatorNegation" is excluded from the mutation run');
+    });
+
+    unaryOperatorMutants.forEach((updateMutant) => {
+      expect(updateMutant.status).eq('Ignored');
+      expect(updateMutant.statusReason).eq('Ignored because the operator "UnaryMinOperatorNegation" is excluded from the mutation run');
     });
   });
 

e2e/test/ignore-project/verify/verify.js.snap

@@ -3,20 +3,20 @@
 exports[`After running stryker on jest-react project should report expected scores 1`] = `
 Object {
   "compileErrors": 0,
-  "ignored": 32,
-  "killed": 8,
-  "mutationScore": 50,
+  "ignored": 34,
+  "killed": 6,
+  "mutationScore": 42.857142857142854,
   "mutationScoreBasedOnCoveredCode": 100,
   "noCoverage": 8,
   "pending": 0,
   "runtimeErrors": 0,
   "survived": 0,
   "timeout": 0,
-  "totalCovered": 8,
-  "totalDetected": 8,
+  "totalCovered": 6,
+  "totalDetected": 6,
   "totalInvalid": 0,
   "totalMutants": 48,
   "totalUndetected": 8,
-  "totalValid": 16,
+  "totalValid": 14,
 }
 `;