GEN0305 uss_qualifier checks to verify ASTM F3548-21

[BenjaminPelletier]

As part of #274, we should check GEN0305.

One possible approach:

Ask for versioning endpoints for both test env and deployment env (accept resources for "test env versioning API" and "deployment env versioning API"), query both and compare. When a group test is conducted, it should be ok for one participant to have a different version ("except when testing an update").

[Shastick]

We were discussing this topic with @mickmis and @barroco last week and ended up with these questions:

Implementation wise, would this look like an additional endpoint that returns the version currently deployed in the test environments as well as the version returned in production (ie, a single endpoint that returns two versions),

or: would each environment have an endpoint that we can query and which returns a single version? (I believe we are not supposed to have any interaction with the production environment, but I'm not entirely sure).

A separate question: should we impose any format on the version (such as SemVer or CalVer). My guess is we should not (the question arised from wondering how we can know when the prod and test version may be different, see below)

Finally, how do we determine when versions are allowed to be different? You mention "group test": how is this different from a 'normal' test? Is this easy to determine? Does it require the test operator to set a particular flag in the configuration?

To solve this I was wondering if the endpoint that returns the version(s) could also include a boolean in the result that tells the qualifier "a new version is being tested".

Draft Proposal

From my current understanding, here is a simple proposal for a new endpoint, which could be added to the test harness:

GET <test_harness_api_base>/versions:
returns {
   test_version: string
   production_version: string
   new_release_under_test: boolean
}

Retained Approach

Discussed at the weekly meeting on 16th of Jan 2024:

• This is a cross-environment requirement, for which we are allowed to interact with the production environment
• We expect to have a single participant testing a new release at a time, so we will expect to have at most one USS with a version mismatch. If we see more, we won't be able to mark GEN0305 as passed
• Getting the running version will require a new endpoint