Rule test engine bug fix - sometimes fail to flag not_applicable and undetermined #1422
Conversation
…et flagged if a ruletest section had exclusively failures of those type.
|
Hi @jugonzal07 I am going to get started addressing the test case failures. I will start another branch with a draft PR and we can merge that branch into this one once complete to hopefully pass the CI |
|
@jugonzal07 Take a look at the CI for the testing of rules 5-5, 5-7, 5-9, 5-11 in this PR. We are not getting any FAILURE messages, it just doesn't say "All tests passed!" at the conclusion of the rule's tests. Can we get these scenarios to raise a more obvious error?
|
…ndetermined_fix' into RT/JDJ/resolve-na-undetermined-failures # Conflicts: # rct229/ruletest_engine/ruletest_engine.py
|
Very weird! I will definitely take a look. thanks @JacksonJ-KC |
…ctionary with an empty list of results, where no indication of outcome is given. This will now correctly log this error rather than failing, but producing no context on why it fails. Also, added additional check in case some other combination of factors results in a failure without a log describing why it fails.
…ndetermined_fix' into RT/JDJ/resolve-na-undetermined-failures
|
I found the issue and added a correction. In short, it looks like these tests are evaluating to nothing at all and return a blank list. This raises failures thankfully, but results in an empty log. My new commit will now correctly log this error. Also, added additional check in case some other combination of factors results in a failure without a log describing why it fails-- at least some log message will let us know. |
|
Perfect, thanks! |
|
Many rules in sections 5 and 22 use list filter to figure out if a rule is applicable. If list filter returns an empty list, then the rule is not evaluated at all and the NOT_APPLICABLE outcome is not actually being received. I think (for rule tests) if context_list is empty then we should continue to provide the FAILURE message that you added, but if filtered_context_list is empty we should provide NOT_APPLICABLE outcome. If it's possible for this only to happen in rule tests then I think that would be preferable. I like that the rules simply don't evaluate when they don't apply to real projects. Rules I have noticed that behave this way: |
…ix' into RT/JG/ruletest_na_undetermined_fix
RS/JDJ/22-17, 22-18, 22-28 Applicability
…tem from being assigned to a multiple zones. This allows us to test cases when a user might accidentally do this. Also, potentially fixes a particular case where multizone systems did not always have the correct system assigned to their terminals (saw it in an edge case).
RT/JDJ/19-3 delete invalid test jsons temporarily
…om/pnnl/ruleset-checking-tool into RT/WX/fix_rule_18_2
RS/JDJ/std equal bugfix
…would be shallow copies across the user, proposed, and baseline.
…-checking-tool into RT/WX/fix_rule_18_2
Rt/wx/fix rule 18 2
Addressed issue where
undeterminedandnot_applicabletests would not get flagged if a ruletest section had exclusively failures of those type. NOTE-- this will result in needing to revisit test cases that passed when they should have failed.