Additional information and better logic for the acceptance test feature

[isabelle-dr]

The "acceptance test" has been created to inform the developer of the impact a PR on the datasets available on the Mobility Database, in order to:

1. for the developer: make sure the code logic is working properly (if the code added is a rule with a WARNING severity and the acceptance test is flagged, there is a problem with the code)
2. for the community: make an informed decision on what to do if the new rule is truly resulting in previously valid datasets becoming invalid. This will depend on the results of the acceptance test and it could be: (i) adding a grace period for producers to fix their feed, or (ii) changing the specification to reflect practice.l

The first version of this feature (built in PR #848, included in the 3.0.0 release) allows our users to:

• at each commit, see a GH action if the code written results in more than 1% of datasets containing a new error occurrence.
• in the GH action comment, see:
  • how many datasets did the test run on in total
  • how many datasets are corrupted
  • a downloadable acceptance_test_report, containing the list of dataset ID's that contain an additional error, the error type responsible, and the number of occurrences of that error in each dataset
  • a downloadable corrupted_sources_report containing the list of corrupted dataset ID's (meaning we don't know how the PR impacts them)
• a specific commit message [acceptance test skip] can be used to prohibit this test from running (if fixing a typo, working on documentation)

Outcome

We evaluated that in order to provide developers with all the information they need, the following changes need to be added to this feature. This list will be updated as we gather more feedback on this feature.

• as a developer writing a new rule, I want to know if my PR results in one additional occurrence of the error type
  • Modify the logic for the acceptance tests #1076 (remove the 1% threshold and trigger the acceptance test for all new occurrences of an error type)
• as a developer modifying an existing rule, I want to know
  • how many datasets were already containing the notice emitted by this rule
  • how many NEW datasets would contain the notice if my PR gets merged
  • how many datasets would NOT contain the notice anymore if my PR gets merged
  • as a result, how many datasets would contain the notice in total if my PR gets merged
  • which datasets are going from valid to invalid if this PR gets merges?
  • which datasets are going from invalid to valid if this PR gets merged?

How will this work?

This additional information could be shared by providing a Google Colab notebook with code snippets to run to get these analytics, or by changing the acceptance test report.

Note

The architecture of the Mobility Database is currently under reconsideration, and since this feature depends on it, we will wait until a decision has been made (February 2022) before moving forward with these changes.