Stress Testing

To stress a solution against several test cases it is required to provide a generator and a brute solution. Call ACMX: Upgrade to create a generator and a brute solution from templates. Fill both files with relevant code.

Generator

The generator should be a code, which when executed, prints in the standard output the input file. Generators usually are non-deterministic, they produce different outputs on every run. The generated file will be named gen.in.

Brute solution

After creating the input file, the brute solution is going to be used to generate the expected answer: gen.ans.

Stress

Call ACMX: Stress to stress the main solution against test cases generated by both the generator, and the brute solution. If it fails with some of the generated test case, this test case will be saved as gen.{index}.in/gen.{index}.ans.

The number of executions can be configured at acmx.stress.times. By default it runs 10 times.

Templates

To use custom template for generator edit the setting acmx.template.generatorTemplate with the path to the generator template.

To use custom template for brute solution edit the setting acmx.template.bruteTemplate with the path to the brute solution template.

The name of the created file in the workspace will be the same as the name of the template. A common pitfall is to use the same template file for the main solution and for the brute solution (or in general to use the same template for two different types of files) since one will overwrite the other. Use different names for each file.

Generators and brute solution language will be determined by each file extension, so several languages can be used. They will be compiled and executed using the same rules used for main solution. See Languages section to learn more about it.

Rich generators

There are some tools online to generate test cases. While writing this tutorial I'm thinking about jngen and testlib. They can be used for generators written in c++ within visual studio code if configured properly. The most important step, is to add relevant library to be used in the compile/run process.

Example with jngen

1. Download jngen.h and store it at ~/jngen/jngen.h.

2. Edit c++ compile line at ~/.acmx/languages/cpp.json to check for this library at compile time:

{
    "preRun": [
        "g++",
        "-fdiagnostics-color=always",
        "-std=c++17",
        "-DACMX",
        "$CODE",
        "-o",
        "$OUTPUT",
        "--debug",
        "-I",
        "~/jngen"
    ],
    "run": ["$OUTPUT"],
    "ext": "cpp"
}

Note the last two parameters added to preRun entry.

WARNING: Don't use "~" in the .json file, instead use absolute path.

3. [Optional] Let vscode know about this library, for autocompletion. Check how to do it for c++ in this SO answer.

4. Write generator using the library:

#include "jngen.h"

int main(){
    // Add code using jngen...
}

Remember to use some granular seed, such that the output on consecutive runs are different. Using time with precision at the level of seconds is a bad idea, since usually these tests run under a second.

5. Run stress, and see how tests are generated.

The previous example can be extrapolated for different languages, and not only for generator, but also for checker and brute solution. It can be used for main solution too, but it is trickier, since in competitive programming you should only submit a single, self-contained file that doesn't depend on anything but the standard library of the relevant language. To see how to use custom libraries for main solution see Smart Copy.