Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Run unit test shuffled and timed #20374

Open
wants to merge 1 commit into
base: master
Choose a base branch
from
+2 −0
Open

Run unit test shuffled and timed #20374

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions Makefile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -47,6 +47,8 @@ tidy-full: tools/tidy
.PHONY: unit-test
unit-test:
prove -l -Ios-autoinst/ t/
prove --shuffle -l -Ios-autoinst/ t/
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Add two additional commands in the unit-test target:

1. run the test with --shuffle argument: it randomly change the execution order. Comparing the result with the previous in-order execution can spot test dependency issues.

I think it's a good idea to also run tests shuffled but not within the default workflows which should always be consistent and reproducible. Otherwise people will be confused because suddenly their PR starts failing. As long as unit tests are still executed quickly enough you can consider to run those multiple invocations in post-merge tests

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it's a good idea to also run tests shuffled but not within the default workflows which should always be consistent and reproducible.

Ok, I'm fine with this. Translating it to "code", does it means to create a dedicated Makefile target about it and move the shuffle command there?

quickly enough
I think so, and last command I added somehow try to also keep UT execution time under control (even if at themoment my command only print out execution time and does not set any threshould)

run those multiple invocations in post-merge tests
Could you please elaborate about it? https://github.com/os-autoinst/os-autoinst-distri-opensuse/blob/master/.github/workflows/ci.yml it has both on: [push, pull_request]
Should I move it in one of these github workflow?

$ grep -r push .github | grep -v pull_req
.github/workflows/commit-message-checker.yml:  push:
.github/workflows/deploy.yml:  push:
.github/workflows/shellcheck.yml:  push:
.github/workflows/trufflehog.yml:  push:

Should I just leave it like a makefile target and maybe just document it in the CONTRIBUTE.md file?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

probably best to create a dedicated Makefile target and a dedicated workflow definition if you want to follow that at all.

prove --timer -l -Ios-autoinst/ t/ 2>/dev/null | grep -E "t/.*\.t.*ok[[:space:]]+([0-9]+)[[:space:]]+ms" | sort -nrk 5,5

.PHONY: test-compile
test-compile: check-links
Expand Down
Loading