You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
We should add a "known issue" result type. It would correspond to when a test input for a platform/library matches criteria for an existing bug filed in the upstream project of the platform.
The benefit of having the result type is to better categorize / triage the results of our executor implementations, as we continue to update and improve them. Currently, we have test failures, and those failures could be the result of: the library (unknown bug), the library (known bug), or a deficiency in our executor. Sometimes, a known issue causes a systematic set of many test failures.
For the purposes of our project, it would help instill confidence in our self-assessment of correct executor implementation work to not include known bugs in the set of test failures.
The text was updated successfully, but these errors were encountered:
PR #256 is moving in this direction, identifying some known issues in test failures, errors, and unsupported. It shows these in DateTime format results as orange bars. It also has some classification of details.
Next steps:
build better ways of identifying test results that are known issues.
add mechanism to find the same test across multiple platforms and ICU versions, using the hexhash value.
We should add a "known issue" result type. It would correspond to when a test input for a platform/library matches criteria for an existing bug filed in the upstream project of the platform.
The benefit of having the result type is to better categorize / triage the results of our executor implementations, as we continue to update and improve them. Currently, we have test failures, and those failures could be the result of: the library (unknown bug), the library (known bug), or a deficiency in our executor. Sometimes, a known issue causes a systematic set of many test failures.
For the purposes of our project, it would help instill confidence in our self-assessment of correct executor implementation work to not include known bugs in the set of test failures.
The text was updated successfully, but these errors were encountered: