Non deterministic Test Tracker

Background

SimpleReport engineers recently identified a pain point pertaining to CI/CD: non-deterministic or "flaky" tests.

Non-deterministic tests cannot be guaranteed to behave predictably, and may cause CI/CD failures even if the underlying code is healthy.

Below, please document any such automated tests. Unit tests, integration tests, and end-to-end tests are all relevant to this effort and may all be tracked on this page. Engineers did a flaky-test bug bash session in the 9/7 eng sync. Agenda and notes can be found here.

Tests

A log of flaky tests are kept here

Reproducing

• Java, use Intellij's repeat until failure can help make it troubleshoot
• Javascript, use jest's test.each() to repeat the test many times.

E2E Tests

Periodically our Cypress test user gets locked out and CI runs will fail in bulk with a 401 Unauthorized error. We suspect something is wrong with the way we're programmatically generating our MFA logins during e2e runs but can't be bothered to fix it. In those cases, go into Okta and unlock that user (ask an engineer for the account details) and rerun the test.