Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Firefox e2e tests flakey #472

Closed
tunetheweb opened this issue May 2, 2024 · 4 comments
Closed

Firefox e2e tests flakey #472

tunetheweb opened this issue May 2, 2024 · 4 comments

Comments

@tunetheweb
Copy link
Member

Looks in Firefox Nightly FCP is often quite slow in our e2e tests.

For example this run has this when I turn on debugging:

[0-1] reports the correct value after the first paint {
[0-1]   name: 'FCP',
[0-1]   value: 53[15](https://github.com/GoogleChrome/web-vitals/actions/runs/8923549678/job/24508034241#step:6:16),
[0-1]   rating: 'poor',
[0-1]   delta: 5315,
[0-1]   entries: [
[0-1]     {
[0-1]       name: 'first-contentful-paint',
[0-1]       entryType: 'paint',
[0-1]       startTime: 5315,
[0-1]       duration: 0
[0-1]     }
[0-1]   ],
[0-1]   id: 'v4-[17](https://github.com/GoogleChrome/web-vitals/actions/runs/8923549678/job/24508034241#step:6:18)14651853858-9340183650873',
[0-1]   navigationType: 'navigate'
[0-1] }

A rerun often solves it as you can see later in that log:

[0-1] reports the correct value after the first paint {
[0-1] name: 'FCP',
[0-1] value: 72,
[0-1] rating: 'good',
[0-1] delta: 72,
[0-1] entries: [
[0-1] {
[0-1] name: 'first-contentful-paint',
[0-1] entryType: 'paint',
[0-1] startTime: 72,
[0-1] duration: 0
[0-1] }
[0-1] ],
[0-1] id: 'v4-1714651859617-3386553824299',
[0-1] navigationType: 'navigate'
[0-1] }


But it doesn't seem to solve it for these tests which have been continually failing:
- FCP - 'reports if the page is restored from bfcache'
- LCP - 'reports the correct value on hidden (reportAllChanges === false)'

I hoped it was just some nightly flakiness but doesn't seem to be getting resolved so we might want to comment these out for now for Firefox.

I can't reproduce this on MacOS, nor on linux (Chromebox) so it only seems to happen in GitHub Actions. Possibly related to there not being a display there?
@tunetheweb
Copy link
Member Author

@tunetheweb
Copy link
Member Author

So #476 resolved a lot of the cases for now but this one is still susceptible to failing intermittently due to the Firefox slowness in GitHub Actions:

const assertFullReportsAreCorrect = (beacons) => {
const [lcp1, lcp2] = beacons;
assert(lcp1.value < 500); // Less than the image load delay.

A rerun often fixes it but might take a few attempts. If it gets really annoying we can consider skipping this last line out for Firefox too.

@tunetheweb
Copy link
Member Author

Either way leaving this open to remind us to remove those workarounds once the real fix is released.

@tunetheweb
Copy link
Member Author

Workarounds were removed in #507 so this can be closed

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant