You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
We don't currently run the Dolt BATS test suite on Mac and Windows platforms in a way that ensures regressions for those platforms don't get merged to main. This means platform-specific regressions that our BATS tests could catch can sometimes get released. We saw an example of that today with a windows-specific drop database issue (#7492).
For Unix/Ubuntu, the BATS test suite currently runs on every PR going to the main branch of the dolt repo, and runs in parallel on AWS Lambda, which gives us good coverage of Unix/Ubuntu environments and only takes around 5 or 6 minutes to execute.
For MacOS, the BATS test suite takes 90+ minutes to run on every PR and often had contention with Mac runners which added further latency. We moved Mac BATS tests to run nightly, which will still alert us to failures, but won't prevent those regressions from reaching main and being released.
For Windows, the BATS test suite hasn't been running anywhere for over a year. Due to not being used, these tests have decayed and need work to get them passing again on Windows. Many individual tests also explicitly skip windows, which reduces the coverage of the test suite.
Ideally, we want to prevent platform-specific regressions from being released to customers, while also balancing the need for developers to run tests in PRs in a reasonable amount of time. The Lambda parallelization has worked extremely well for the Unix/Ubuntu tests, but is not as straightforward for Mac and Windows, so another approach my be necessary.
The text was updated successfully, but these errors were encountered:
We don't currently run the Dolt BATS test suite on Mac and Windows platforms in a way that ensures regressions for those platforms don't get merged to main. This means platform-specific regressions that our BATS tests could catch can sometimes get released. We saw an example of that today with a windows-specific
drop database
issue (#7492).For Unix/Ubuntu, the BATS test suite currently runs on every PR going to the main branch of the dolt repo, and runs in parallel on AWS Lambda, which gives us good coverage of Unix/Ubuntu environments and only takes around 5 or 6 minutes to execute.
For MacOS, the BATS test suite takes 90+ minutes to run on every PR and often had contention with Mac runners which added further latency. We moved Mac BATS tests to run nightly, which will still alert us to failures, but won't prevent those regressions from reaching main and being released.
For Windows, the BATS test suite hasn't been running anywhere for over a year. Due to not being used, these tests have decayed and need work to get them passing again on Windows. Many individual tests also explicitly skip windows, which reduces the coverage of the test suite.
Ideally, we want to prevent platform-specific regressions from being released to customers, while also balancing the need for developers to run tests in PRs in a reasonable amount of time. The Lambda parallelization has worked extremely well for the Unix/Ubuntu tests, but is not as straightforward for Mac and Windows, so another approach my be necessary.
The text was updated successfully, but these errors were encountered: