Run unit test shuffled and timed

[mpagot]

Add two additional commands in the unit-test target:

1. run the test with --shuffle argument: it randomly change the execution order. Comparing the result with the previous in-order execution can spot test dependency issues.
2. run test with --timer, parse and order the result to print the test in order of time to run. In can help to spot tests taking too much time as maybe using not "mocked" sleep.

• Verification run:
  https://github.com/os-autoinst/os-autoinst-distri-opensuse/actions/runs/11272220692/job/31346686699?pr=20374#step:6:306

[okurz]

Add two additional commands in the unit-test target:

1. run the test with --shuffle argument: it randomly change the execution order. Comparing the result with the previous in-order execution can spot test dependency issues.

I think it's a good idea to also run tests shuffled but not within the default workflows which should always be consistent and reproducible. Otherwise people will be confused because suddenly their PR starts failing. As long as unit tests are still executed quickly enough you can consider to run those multiple invocations in post-merge tests

[mpagot]

I think it's a good idea to also run tests shuffled but not within the default workflows which should always be consistent and reproducible.

Ok, I'm fine with this. Translating it to "code", does it means to create a dedicated Makefile target about it and move the shuffle command there?

quickly enough
I think so, and last command I added somehow try to also keep UT execution time under control (even if at themoment my command only print out execution time and does not set any threshould)

run those multiple invocations in post-merge tests
Could you please elaborate about it? https://github.com/os-autoinst/os-autoinst-distri-opensuse/blob/master/.github/workflows/ci.yml it has both on: [push, pull_request]
Should I move it in one of these github workflow?

$ grep -r push .github | grep -v pull_req
.github/workflows/commit-message-checker.yml:  push:
.github/workflows/deploy.yml:  push:
.github/workflows/shellcheck.yml:  push:
.github/workflows/trufflehog.yml:  push:

Should I just leave it like a makefile target and maybe just document it in the CONTRIBUTE.md file?

[okurz]

probably best to create a dedicated Makefile target and a dedicated workflow definition if you want to follow that at all.

[foursixnine]

I'm here with @okurz - shuffling makes it harder to figure out what's happening and I fail to see atm the situation with test dependencies (can you enlighten me here? what's the problem you're trying to solve?)

[mpagot]

I'm here with @okurz - shuffling makes it harder to figure out what's happening and I fail to see atm the situation with test dependencies (can you enlighten me here? what's the problem you're trying to solve?)

I was digging in https://perldoc.perl.org/prove and noticed that execution mode. From my experience with other unit testing frameworks, it is a good idea to have UT suite also run in this mode to spot hidden issue in UT implementation (like test 42 works only be chance as test 37 has configured some global variable). I try to probe Test Isolation.