Writing regression tests

RegtestData

A class that holds state information that allows a test to communicate with bytesalad (Artifactory). Attributes point to were input and truth data files live locally and on Artifactory, and where outputs from the test live locally.

RegtestData.input
RegtestData.input_remote
RegtestData.output
RegtestData.truth
RegtestData.truth_remote

Also provides some convenience methods to get the input and truth data from Artifactory into the local directory where the test is running.

RegtestData.get_data()
RegtestData.get_truth()
RegtestData.get_asn()

Fixtures that return a RegtestData instance to the test:

• rtdata returns an instance of RegtestData to a test for use.

• rtdata_module returns an instance of RegtestData to another fixture that can be scoped at function or module level, and then that fixture in turn runs a pipeline or step, and then those results get supplied to individual tests that can test the results of the pipeline or step in various ways.

JSON breadcrumbs get left behind on failure

• As in our current system, if a test fails, a JSON specfile gets left behind, and Jenkins slurps these up and uploads the failed outputs to Artifactory
• Additionally, a JSON specfile that allows OKification gets left behind. We will have to write a small script to utilize these to allow okifying.