[uss_qualifier] Add prod system version checking #532
Conversation
| self._test_env_version_providers = test_env_version_providers.version_providers | ||
| self._prod_env_version_providers = ( | ||
| prod_env_version_providers.version_providers | ||
| if prod_env_version_providers |
There was a problem hiding this comment.
I'm not sure I fully understand the impact of where the ? is in the yaml file. Won't this scenario be skipped with the current f3548_21.yaml file if prod_env_version_providers is not provided? Although there is a ? in the resources required by the suite, there is not in the resources required by the scenario.
There was a problem hiding this comment.
Yeah, this probably warrants more explicit documentation. In the resources section of a test suite definition, a ? means the resource is optional. That is, it's ok to run the test suite even if that resource isn't provided. In a test scenario declaration, it means roughly the same thing: the test scenario can be run even if that resource isn't provided.
When a resource is marked with a ? in a test scenario, the constructor of that scenario must have a default value for the resource (e.g., None). A test scenario in a test suite that can be run will be run. Therefore, if a test scenario wants resource X but resource X is not provided, then the test scenario will be run if X is marked with a ? in the test scenario declaration but not run if X is not marked with a ? in the test scenario declaration.
In this case, the version providers resource may or may not be provided to the test suite -- the test suite can still run effectively if it's not provided. But, if it's not provided, then the test scenario will be skipped entirely rather than trying to run it without that resource.
There was a problem hiding this comment.
OK I see thanks for the clarification.
In this case, the version providers resource may or may not be provided to the test suite -- the test suite can still run effectively if it's not provided. But, if it's not provided, then the test scenario will be skipped entirely rather than trying to run it without that resource.
Then, if the resource is missing, the scenario is skipped, doesn't that mean that prod_env_version_providers will always evaluate as truthy here? Or is it possible for a resource to exist but to be empty?
There was a problem hiding this comment.
Ah, right, sorry -- my mistake; this conditional was left over from when I actually did have the prod version providers as an optional resource earlier in development (it no longer is). Removed -- thanks for the catch
There was a problem hiding this comment.
Ah OK! Not that it's a big deal, but I felt like there was something I didn't get :)
monitoring/uss_qualifier/scenarios/astm/utm/versioning/evaluate_system_versions.py
Show resolved
Hide resolved
* Add prod system version checking * Add note per comments * Remove default value for non-optional parameter * make format 817160d
This PR fixes #467 by adding production/deployed system version checks to the F3548-21 test suite in the form of a new EvaluateSystemVersions scenario. The primary purpose of this scenario is to query production and test systems for their versions and compare those values. If more than one participant has a version mismatch, then the check fails (one participant can have a mismatch as this indicates they are testing a new software version). This scenario necessitates the provision of an additional resource specifying how to contact the production system to query its version.
An additional check is also added to look back through past version queries during the test run (if there are any, which there usually are for F3548-21) and makes sure the software version of the system under test didn't change during the test run.