Make Unit Test Great!

[HuSharp]

Development Task

• The unstable CI has been a hinder to our daily development. Sometimes we have to run it again and again, and again ... to merge a PR.
• When run make basic-test in local machine, there will be a lot of strange problems, as well as running extremely slow :(

Need 5:44.15 total for Intel(R) Xeon(R) Gold 6240 CPU @ 2.60GH, 144 core 72 CPU
but local mac(m1pro 16G) needs > 15min

We use a wrong way to parallel. Some test cases are not side effect free, for example:

• the enable and disable of the failpoint
• modification of the configuration
• change of the global variable in a single process.
  etc...

What's the solution?

Add back parallel by run a single test case in a isolated OS process #7952

If the test cases with side effect run parallelly with others, it could cause some test fail unexpectedly.
When they're in a single OS process, make them parallel is not a big deal.

print function time

when using ./pd-ut run --junitfile xxx, can get junitfile as XML file to get every function cost time.

Furthermore

• handling the coverage.txt & integration with the CI script *: support generate the test coverage file and using it in ci #7991
• classify some real integrated tests and move them to the tests directory Sort out the unit tests #7930
• support cache mechanism to accelerate bin build tools/pd-ut: accelerate build by compile-without-link #8069
• change required ci config(pd): update pd master required context ti-community-infra/configs#1048
• support running integration test tools/ut: replace integration test in ci #8200
• support parallel parameter tools/ut: add a parallel parameter #8186
• unify tests HTTP client tests/client: unify http client in tests #8199
• support an alloc server to alloc URL tools/ut: add alloc server for pd-ut #8203

Related

• Ref TiDB Make CI great again! pingcap/tidb#30822
• Enhance the integration test of CI Enhance the integration test of CI #7298

Result Show

Local Test

Since we run parallel now, the CI runs faster than before, on my server computer (Intel(R) Xeon(R) Gold 6240 CPU @ 2.60GH, 144 core 72 CPU):
reduce from 6min -> 1min

for make ut:

time make ut
building task finish, parallelism=144, count=641, takes=10.154869881s
run all tasks takes 50.559951196s
> make ut  327.15s user 48.61s system 615% cpu 1:01.03 total

for make basic-test

time make basic-test
CGO_ENABLED=1 go test -tags tso_function_test,deadlock -timeout 20m -race
> make basic-test 1176.46s user 169.41s system 473% cpu 5:44.15 total

CI Result

Before	After