Scheduled testing does not specify test version

[aidanheerdegen]

This failure of the scheduled testing

ACCESS-NRI/access-om2-configs#41

highlights that the scheduled testing doesn't define a version of the tests to run. This makes it fragile to changes in the test infrastructure, and so makes it more difficult to change the tests as they could potentially affect many repro tests of older configurations.

Related issues: ACCESS-NRI/access-om2-configs#41

[aidanheerdegen]

Ping @CodeGat

[CodeGat]

For this feature, I was thinking about just adding a new field to the released-configs.json where we could specify a commit-ish object that points to the tests.
Would just a commit hash be good enough to add to that file, or do we want to start versioning tests via tags in this repository as well? We do kinda demarcate the configs by releases now (and tags) so it might not be as much of an issue. But if we start bundling tests into packages properly we might not want to start tagging versions of tests...

[aidanheerdegen]

Heh. Should have read this comment before I made this one:

ACCESS-NRI/reproducibility#26 (comment)

Yes I am wondering how this will work with packaged tests.

[CodeGat]

It might need a small amount of rejigging, but we will still need something very similar to this when the tests are packaged up, anyway

[CodeGat]

Closed the linked PRs because it was a bit of a half-measure towards the test packaging stuff

[aidanheerdegen]

As discussed IRL, but wanted to make a note, the tests version would need to be defined for each tag. It could just be main (or some default pointing to latest when they're packaged), but it would mean we have the option of pinning to a version if tests start failing because they've been updated, but the version of the model is inconsistent with the updated tests.

It could (will) end up creating a support burden to have to update lots of historical configurations to be consistent with updated tests.