[uss_qualifier] f3548 - GEN0300 verify interactions with interoperability test instance

[Shastick]

Extends the f3548 aggregate check to look for queries that interact with the test instance in order to validate requirement n° GEN0300.

If no interaction is found, the check is skipped.

Otherwise, the check is reasonably lax: for each possible interaction with the test endpoint, it is enough to have a single successful query to pass.

[Shastick]

This is pretty lax, but I was wondering it it is reasonable to expect all queries to succeed: if there's a stray network failure/timeout that gets taken care of through a retry it would probably be too stringent to mark the requirement as failed.

[BenjaminPelletier]

I think that's fine, and I do agree that requiring all queries to succeed would be going to far (I don't think we should conclude that a test instance wasn't being provided if 99% of queries to it succeeded). I think our detection of a violation of the requirement to provide a test instance should be pretty robust with this implementation as-is.

[Shastick]

@BenjaminPelletier this approach is slightly different than what I remember discussing ~10 days ago (which I wrote down as "find a check that interacts with the test instance and mark GEN0300 as checked").

As I ran into the QueryType constants that seem to include the interactions with the test instance I opted to add a test case to the aggregate checks at the end.

[BenjaminPelletier]

@BenjaminPelletier this approach is slightly different than what I remember discussing ~10 days ago (which I wrote down as "find a check that interacts with the test instance and mark GEN0300 as checked").

As I ran into the QueryType constants that seem to include the interactions with the test instance I opted to add a test case to the aggregate checks at the end.

This seems like a reasonable approach to me, and probably superior to what I was originally thinking. GEN0300 says the USS needs to provide a test instance and I think this approach effectively identifies when that requirement isn't met with a fairly small amount of implementation.

[BenjaminPelletier]

LGTM. Not merging yet because I think main needs to be merged back into this dev branch and make format run to avoid CI complaints when merged to main.

[BenjaminPelletier]

I'm not sure I follow this; what would be removed at completion of development?

[Shastick]

I was referring to the test_interactions == 0 condition and the call to record_failed that is guarded by it: given that if the condition is true we returned above already.

(The condition was there to fail noisily while I was building the scenario)