Implementation of mutation levels

[Ja4pp]

What?

We've added support for mutation levels to improve the performance of Stryker. It includes mutation operators, mutation groups, mutation levels, documentation and tests. In addition, we have PRs for the reporters in stryker-mutator/mutation-testing-elements and stryker-mutator/stryker-dashboard. Those PR's are still a work in progress.

Why?

Currently, Stryker, by default, executes a comprehensive mutation testing suite on your project, offering maximum efficacy. However, this default setting demands substantial computational resources. Alternatively, users can leverage mutation levels for a more resource-efficient testing experience, ranging from level 1 (optimized for speed) to level 3 (prioritizing efficacy). For more background information, see the Calisto research (https://research.infosupport.com/callisto/).

How?

For a tailored configuration, Xavier allows the adjustment of mutation levels by specifying inclusion and exclusion fields. In the configuration file, mutation operators can be precisely designated by their literal names, while mutator groups and mutation levels are denoted with the @ prefix (e.g., @ArithmeticOperator or @level1). For instance, to refine level 2 by selectively retaining the AdditionOperatorNegation mutation operator and excluding all other operators within the ArithmeticOperator group, the configuration would be expressed as follows:

{
   "includedMutators": ["@level1", "AdditionOperatorNegation"],
   "excludedMutators": ["@ArithmeticOperator"]
}
   

Testing?

We have made modifications to the ignore-project in the e2e test folder to support mutation levels as it had excluded mutations already in its configuration file. Moreover, we added unit tests for all mutator operators. We tested a subset of operators, an empty level and an undefined level.

[nicojs]

Looks a lot better already! The biggest remark I have is about the NodeMutator API.

Thanks a lot for the work so far 👏

[nicojs]

I think it's a good idea to separate this config, except I would expect it to be in a .ts file, so you can simply import it, instead of having to load it some other way.

Especially since the names used here should be in sync with the names in the stryker-core.json file. Having it as a JSON file here breaks type safety.

[dvcopae]

Good observation! We moved the default-mutation-levels to a .ts file, and used the spread operator to remove some redundancy between the levels.

[nicojs]

Instead of having each NodeMutator decide on whether to generate a mutant, it would be cleaner if they always generate everything and let excluding be done afterward (in the BabelInstrumenter class). It would safe you a lot if-statements and features like "absolute mutant score" would be easier to implement later.

[dvcopae]

Implemented, this indeed leaves a lot of if statements and duplicate code out.

[nicojs]

TODO: decide on the actual names

[dvcopae]

We have also updated the supported-mutators documentation here, based on your PR .

We also noticed a few mutation operators that were not in the documentation as well, so we tried to come up with new names for those following your convention.

[nicojs]

I'm not sure if an any is needed here. I think it was strongly typed before.

[dvcopae]

Updated the type to Node | Node[] | boolean | string | null

[nicojs]

I think a better name is "mutationOperator" (in line with research papers)

[nicojs]

It is very useful to see all the descriptions here. It will help Stryker users enormously. 👏

[dvcopae]

Changes since Nico's last review:

• Moved default-mutation-levels.js to a .ts file and used the spread operator to make it clearer that one level extends the other
• Replaced mutationName with mutationOperator
• Removed any type from ReplacementConfiguration
• Removed everything related to the clear-text reporter changes
• Enforce a minimum array size of 1 for the includedMutations property
• Print a warning when something is not recognised in the includedMutations/excludedMutations configuration
• Extend documentation about the concept of mutation levels, and how Callisto can be used to make adjustments to the default mutation levels