-
Notifications
You must be signed in to change notification settings - Fork 19
Automated Testing: current status and TODOs
Status as of May 11th 2023.
Covered by tests:
- GEN0310
- GEN0500
- OPIN0015
- OPIN0020
- OPIN0025
- OPIN0030
- OPIN0040
- USS0005
- USS0105
- SCD0015
- SCD0020
- SCD0025
- SCD0030
- SCD0035
- SCD0040
- SCD0045
- SCD0050
Probably not automatically testable:
- OPIN0005
- OPIN0010
- OPIN0035
- Define (through automated testing interfaces most likely) a mechanism enabling the test driver to validate notifications emitted by USSs (e.g. upon new conflicts) to their own operators or to other USSs.
- Add state check
- Update according to comment from @BenjaminPelletier:
But I think the intent of the original check, if I remember correctly, is to make sure the operational intent actually shared is compatible with the user's flight intent. So, if the user requested a flight in the park but the operational intent didn't cover the park, this check would fail. This is a little tricky, however, as there isn't a requirement for a USS to exactly/tightly wrap a user's flight area when creating an operational intent, so the check can't be simply that the op intent volumes equal the injected user intent volumes. Certainly there should be no part of any user intent volume that is outside the union of the shared operational intent volumes, and there are probably other reasonable constraints we can check (it wouldn't be reasonable to make the op intent too much larger than the user intent, for instance), but I think we still do want to have that kind of check, and I think this is the place we would do that.
Ideally, we would have already written this down in the ExpectedBehavior requirement, but I haven't had a chance to think carefully to do that yet. I do want to do that, or I'd welcome suggestions that we can iterate on.
As per comment from @BenjaminPelletier (original):
We can attempt to test for near-intersections, but we have to carefully consider whether each response is valid, and whether each response validates requirement compliance. In the case of instructing a USS to plan a flight with a user op intent that is very close to an existing op intent but not touching, if the USS does plan that flight and the shared op intent is consistent with the request (encompasses all requested area at least), then we can conclude the USS meets that requirement. If the USS declines to plan the flight, we cannot conclude they failed that requirement nor can we conclude that they meet that requirement under the set of circumstances tested. In that case, we would probably want to invent a separate test to cover that requirement.
For instance, the buffering USS could be instructed to plan a flight with no other flights in the area, which they would do successfully (after buffering the user flight intent to construct a larger shared op intent). Then, uss_qualifier could have a mock USS or other USS, which would use op intents exactly as provided by the user, plan a flight right next to the first USS. Then, the first USS could be instructed to activate their flight. Any buffer was already applied, so presumably this should succeed unless the first USS incorrectly comprehends the close-but-not-intersecting op intent as intersecting. It's possible there might be some USS implementation that would grow a user op intent more when activating than when planning, but I'm not thinking of a good reason to do that, and certainly the class of USSs that would do that is smaller than the class of USSs that would grow a user op intent when planning.
- Use instead the structure defined in
uas_standard
Open questions:
- How do we get to have a down USS? Should the test driver be able to instruct a USS to declare itself as down?
- Should the requirements be tested through carefully crafted scenarios (if that is possible), or should the test driver be able to see how the tested USSs are processing operational intents of other USSs?
- Define and implement scenario Nominal planning: permitted conflict with equal priority in
monitoring/uss_qualifier/scenarios/astm/utm/nominal_planning/conflict_equal_priority_permitted/ - In order to test the bulk of what this scenario does, the mechanism for validating notifications is needed.
- Given that whether equal priority conflicts are permitted (and/or for which
priority levels) is defined by the jurisdiction, it should be possible to not
execute the scenario if
Noneis provided as aFlightIntentsResource, just like its sibling scenario Nominal planning: not permitted conflict with equal priority. - Currently, the test USS with the
CHElocality is configured to not permit equal priority conflicts (allows_same_priority_intersections).
A relevant feedback from @BenjaminPelletier:
Whether conflicts are allowed between same-priority operational intents is a property of the jurisdiction rather than each participant. If, e.g., Switzerland said that conflicts are not allowed between normal-priority flights, nor are conflicts allowed between special-priority flights (this is what the U-space regulation appears to require), then it wouldn't make sense for any participant to support conflicts between flights of either of those priorities.
I think the way we'd want to represent that structurally is that the test designer (effectively acting as an agent of the jurisdiction) can design the test to either test for allowed conflicts between two same-priority flights, or not test for that. In order to test for allowed conflicts between two same-priority flights, uss_qualifier would need a set of flight intents with an allowed conflict between them. If the test designer doesn't provide this resource, I think that's a fair way of indicating that the jurisdiction doesn't allow conflicts between two flights of any priority (like U-space). So, for the same-priority conflicts-allowed scenario, we can just accept an Optional[FlightIntentsCollection] resource (see scenarios.astm.netrid.NominalBehavior for an example where dss_pool is an optional resource) and if that resource isn't provided, we simply do nothing in the test (perhaps add a note to the report that the jurisdiction doesn't include any same-priority conflicts-allowed situations).
As a secondary note, it isn't super clear in some places in the standard (particularly SCD0035, SCD0040, SCD0045, and SCD0050), but the intent is that whether the or not conflicts are allowed is on a per-priority-level basis -- that's clearer in SCD0055, SCD0060, SCD0065, and SCD0070 with their "that same priority level" phrasing rather than "the same priority level" of the previous four requirements. So, e.g., a regulator can allow conflicts for priority 10 but not allow conflicts for priority 0.
- It was suggested by @barroco that the system should be put under stress in order to test those requirements in a meaningful way.
- Add a simple scenario where a USS not supporting the CMSA role is instructed to transition an operational intent to Nonconforming and to Contingent states.
- SCD0080
- SCD0085
- SCD0090
- SCD0095
- GEN0100
- GEN0105
- GEN0300
- GEN0305
- GEN0400
- GEN0405
None are tested at the moment:
- LOG0005
- LOG0010
- LOG0015
- LOG0020
- LOG0025
- LOG0030
- LOG0035
- LOG0040
- LOG0045
- LOG0050
- Write requirements doc in
monitoring/uss_qualifier/requirements/astm/f3548/v21/cmsa.md
- Write requirements doc in
monitoring/uss_qualifier/requirements/astm/f3548/v21/cstm.md
- Write requirements doc in
monitoring/uss_qualifier/requirements/astm/f3548/v21/cstp.md