Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

GEN0100 and GEN0105 uss_qualifier checks to verify ASTM F3548-21 #468

Closed
BenjaminPelletier opened this issue Jan 11, 2024 · 2 comments
Closed
Assignees
Labels
automated-testing Related to automated testing tools enhancement New feature or request P1 High priority

Comments

@BenjaminPelletier
Copy link
Member

BenjaminPelletier commented Jan 11, 2024

As part of #274, we should check GEN0100 and GEN0105. Presumably these requirements will be checked together since a good way of measuring time synchronization (GEN0100) is via timestamps (GEN0105).

Possible approach 1:

When op intent details change (creation or update), note when the change was made. If volumes end time is before time change was made, USS may be violating USS0105, or it may be violating GEN0100, or it may be violating GEN0105.

Possible approach 2 (not mutually exclusive):

If/when we obtain log data, compare timestamps for when we know certain events occurred to when they are indicated in the logs1, 2, 3

@BenjaminPelletier BenjaminPelletier added enhancement New feature or request automated-testing Related to automated testing tools P1 High priority labels Jan 11, 2024
@BenjaminPelletier BenjaminPelletier changed the title GEN0100 uss_qualifier checks to verify ASTM F3548-21 GEN0100 and GEN0105 uss_qualifier checks to verify ASTM F3548-21 Jan 11, 2024
@Shastick
Copy link
Contributor

Shastick commented Jan 15, 2024

Discussing the topic with @mickmis and @barroco, we were wondering if this might not require adding a new endpoint, depending how strictly we want to test the requirements:

The gist of our discussion:

  • do we do loose testing by any means (direct or indirect) available?
    • indirect:
      • your suggestion is such an approach.
      • possibly we can make assumptions on the default values used by the DSS when we omit to specify the start time of a volume
      • we could add an endpoint returning the current time of the DSS
      • we could update some/all endpoints for them to include the current time of the DSS
  • do we do strict testing, by checking that anything or most of the entities that get timestamped by the DSS receive the correct timestamp? A very strict interpretation of GEN0105 would probably require that

We also conclude that logs will likely contain what we need to cover GEN0100/0105, but these won't be available for the initial MVP so we need an alternative.

(If this requires more discussion/editing we can take it to a separate design doc)

Retained approach

At the weekly meeting held on 16th of Jan '24, we agreed to have a first check using the "create something in the past" approach, which should be forbidden, and expect that the USS will reject it.

More precisely, we would call the notifyOperationalIntentDetailsChanged endpoint of a USS to create and then modify an operational intent so that it ends in the past, something that the USS should not allow if it implements the endpoints correctly and is using a correct time-source.

A more comprehensive coverage of GEN0100/105 will be done once we have access to log data.

@mickmis
Copy link
Contributor

mickmis commented Mar 28, 2024

Implemented in #555

@mickmis mickmis closed this as completed Mar 28, 2024
@github-project-automation github-project-automation bot moved this from In progress to Done in Basic SCD automated testing Mar 28, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
automated-testing Related to automated testing tools enhancement New feature or request P1 High priority
Projects
Status: Done
Development

No branches or pull requests

3 participants