Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support a "known issue" test result type #226

Open
echeran opened this issue May 6, 2024 · 2 comments
Open

Support a "known issue" test result type #226

echeran opened this issue May 6, 2024 · 2 comments
Assignees

Comments

@echeran
Copy link
Collaborator

echeran commented May 6, 2024

We should add a "known issue" result type. It would correspond to when a test input for a platform/library matches criteria for an existing bug filed in the upstream project of the platform.

The benefit of having the result type is to better categorize / triage the results of our executor implementations, as we continue to update and improve them. Currently, we have test failures, and those failures could be the result of: the library (unknown bug), the library (known bug), or a deficiency in our executor. Sometimes, a known issue causes a systematic set of many test failures.

For the purposes of our project, it would help instill confidence in our self-assessment of correct executor implementation work to not include known bugs in the set of test failures.

@echeran echeran added this to the 2024 Q2 ⟨P1⟩ milestone May 6, 2024
@sven-oly
Copy link
Collaborator

sven-oly commented Aug 8, 2024

PR #256 is moving in this direction, identifying some known issues in test failures, errors, and unsupported. It shows these in DateTime format results as orange bars. It also has some classification of details.

Next steps:

  1. build better ways of identifying test results that are known issues.
  2. add mechanism to find the same test across multiple platforms and ICU versions, using the hexhash value.

@sven-oly
Copy link
Collaborator

Progress made with known issues in Date/Time formatting in PR #329, handling "alternate" vs. "standard" formatting results.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants