Rework MRI test exclusions #3658

…s can cause false positives. We might process two tests named `test_rstrip` and `test_rstrip_bang`. Without the delimiter as part of the prefix, `test_rstrip` would look like a prefix for `test_rstrip_bang`, causing the tool to skip `test_rstrip_bang` thinking it was already added. Depending on the order the tests were processed in you could get two different sets of tags with the same test failures. This was causing us to lose some tags and get different results when retagging MRI tests with random seeds.

The old logic emitted the tags in the order the tests were executed. As a result, if tests are run with random seeds you can get two different files.

… hang. While this mode means we may continue to exclude a test for an issue that may be fixed, the common problem is the test continues to hang. With the tag file completely cleared we attempt to run that test again and it halts the entire process, require manual intervention to: * Kill the stuck process * Manually re-add the tag * Re-run the tests and pipe into the error processing script without the hepl of `jt tag` in order to run the tests with the partially filled tag file

In some cases we have directories that match the search prefix and they will naturally not include the tagged test data.

…e for a tagged test.

Some tests deliberately insert broken bytes or the test sets the encoding to something other than UTF-8. In those cases we may have strings with byte sequences in them that are invalid in UTF-8. Since these are limited to error messages, we can safely ignore them. What we really need to be able to do is split each line in the results file apart and extract key pieces of information. The method names, which are most import for setting up any test exclusions, must be made up of valid bytes.

Scrubbing isn't an entirely fool-proof method, but all of the class names thus far have been ASCII-only, and that's really the data we're after. Non-UTF-8 byte sequences elsewhere in the test will be mangled but we're not doing anything with them here. When we run the tests they'll execute with the correct encoding.

…s raised but with the wrong message.

We tend not to retag the MRI tests very frequently. Having the test Ruby version information there gives an idea of when the test was last run and what build type was use (e.g., native vs JVM). It's easy to ping-pong on retagging slow tests because they're necessarily dependent on the version of TruffleRuby and the system the tests were run on. Providing that context should help us keep track of what should be reevaluated and when a slow tag should be retained."

We currently run the MRI tests with assertions enabled. Those assertions all escape the interpreter and result in the process failing. That can result in the existing tag file getting deleted without being repopulated. What we do here is capture stderr along with the test log and parse it to see if we have any escaped exceptions. If so, tag the corresponding test and then tell the retagger to rerun the test. The new tag will be retained so that test will not run again, allowing the testing process to make forward progress. This makes retagging more reliable and less frustrating.

…ed symbols. When we encounter the usage of an undefined symbol the process exits, which means we don't end up running all of the tests. Much like we do with assertion errors, this will now tag the associated test and then restart the tagging operation, retaining the newly created tags so the associated tests will be skipped.

…rash.

…killed while running.

When a test times out it often will not print a trace. The previous logic would try to extract one anyway and that could result is misattribution since the data format doesn't match what the search logic is looking for. Rather than search for something that isn't there, we now can tag the test with a message indicating the test timed out and move on with processing the next failure or error.

…orm-specific.

…r to understand.

…MRI test log file. Also reorganized the code a bit since the comments are long and it was breaking up the flow of the code.

With enhancements to the retagging process we now have the ability to generate much more helpful reasons for each tag. This change replaces most "needs investigation" generic comments with something that makes it much easier to see why an MRI test failed. Additionally, we had many stale tags, which have now been deleted as part of the bulk retagging process.

Now that we have robust retry logic on errors that bubble out and stop the test runner process, we don't need to retain those tags; they can be safely recomputed. Other patterns are just obsolete and should have been removed when the last reference to them was removed in the tag files.

…inst the test name.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Rework MRI test exclusions #3658

Rework MRI test exclusions #3658

Commits on Sep 24, 2024