wpt.fyi results intermittently conflict between current and history view

[cookiecrook]

Starting with WPT.fyi since it's conveying conflicting results... I'm not sure if that's a data source problem from the wpt repo, or a bug here.

WPT.fyi lists a Safari failure on subtest "[xlink:title][href] > rect" but the Safari history shows it passing always, which aligns with our local testing... The conflicting results listed on the same wpt.fyi page can't both be true.

https://wpt.fyi/results/svg-aam/name/comp_host_language_label.html?label=master&label=experimental&aligned&view=interop&q=label%3Ainterop-2024-accessibility

Therefore I am assuming this is an error in how WPT.fyi tracks the results, rather than in the actual result.

@gsnedders hypothesized "The history view might just be looking at the first/last run of each day?"

[cookiecrook]

FWIW, this is no longer showing an error immediately after filing. 🤷

[cookiecrook]

Sometimes seeing it here, too.
https://wpt.fyi/results/accname/name/comp_labelledby.html?label=master&label=experimental&aligned&view=interop&q=label%3Ainterop-2024-accessibility

[cookiecrook]

That one's also no longer reproducing.

[cookiecrook]

Proof the current result sometimes shows as failing, even though the history view never shows failures. This bug is not tracking the specific failure... It's tracking the fact that WPT.FYI sometimes shows conflicting results on the same page.

[past]

@DanielRyanSmith is this expected? I think the history view captures a single daily run, so it may have missed some intermittent failures during the day, if I understood correctly what you told me recently.

[DanielRyanSmith]

@gsnedders hypothesized "The history view might just be looking at the first/last run of each day?"

@DanielRyanSmith is this expected? I think the history view captures a single daily run, so it may have missed some intermittent failures during the day, if I understood correctly what you told me recently.

These is essentially correct, as the history timeline is populated by finding the first set of aligned runs that occur each day. It's possible that it will not find a flake if the flake it happening rather rarely.

Additionally, I'm seeing in the history timeline now that there are some runs that occasionally crash for this test, although this might be unrelated.

https://wpt.fyi/results/svg-aam/name/comp_host_language_label.html?label=master&label=experimental&max-count=10&product=safari&q=seq%28%28status%3APASS%7Cstatus%3AOK%29%20%28status%3A%21PASS%26status%3A%21OK%26status%3A%21unknown%29%29%20seq%28%28status%3A%21PASS%26status%3A%21OK%26status%3A%21unknown%29%20%28status%3APASS%7Cstatus%3AOK%29%29