Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve BATS test execution for Mac and Windows #7498

Open
fulghum opened this issue Feb 15, 2024 · 0 comments
Open

Improve BATS test execution for Mac and Windows #7498

fulghum opened this issue Feb 15, 2024 · 0 comments
Labels

Comments

@fulghum
Copy link
Contributor

fulghum commented Feb 15, 2024

We don't currently run the Dolt BATS test suite on Mac and Windows platforms in a way that ensures regressions for those platforms don't get merged to main. This means platform-specific regressions that our BATS tests could catch can sometimes get released. We saw an example of that today with a windows-specific drop database issue (#7492).

For Unix/Ubuntu, the BATS test suite currently runs on every PR going to the main branch of the dolt repo, and runs in parallel on AWS Lambda, which gives us good coverage of Unix/Ubuntu environments and only takes around 5 or 6 minutes to execute.

For MacOS, the BATS test suite takes 90+ minutes to run on every PR and often had contention with Mac runners which added further latency. We moved Mac BATS tests to run nightly, which will still alert us to failures, but won't prevent those regressions from reaching main and being released.

For Windows, the BATS test suite hasn't been running anywhere for over a year. Due to not being used, these tests have decayed and need work to get them passing again on Windows. Many individual tests also explicitly skip windows, which reduces the coverage of the test suite.

Ideally, we want to prevent platform-specific regressions from being released to customers, while also balancing the need for developers to run tests in PRs in a reasonable amount of time. The Lambda parallelization has worked extremely well for the Unix/Ubuntu tests, but is not as straightforward for Mac and Windows, so another approach my be necessary.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

1 participant