CI job for verifying coverage increase

[dergoegge]

We could consider adding a CI job that checks that the coverage for newly added inputs actually goes up (this should be possible with afl-showmap). This would help with review and avoiding bloat in the corpora.

(Fuzz harnesses with low stability could be annoying here)

[maflcko]

DrahtBot can (manually triggered) create html coverage reports. See https://github.com/maflcko/DrahtBot/blob/main/coverage_fuzz/src/main.rs#L151 and https://drahtbot.space/host_reports/DrahtBot/reports/coverage_fuzz/monotree/004367dba8a3e852/428a2e7b0def5102/fuzz.coverage/index.html

libFuzzer could also be used by using the ft metric, printed from the summary. Example:

#1206	DONE   cov: 6161 ft: 36449 corp: 941/15703Kb lim: 446621 exec/s: 0 rss: 658Mb

[maflcko]

Fixed in bitcoin/bitcoin#29329

[dergoegge]

I presume the plan is to use that for a new CI job here? i.e. run once for both qa-assets from main and the pr, then compare coverage summary for relevant harnesses and if the coverage does not go up fail the job

[maflcko]

I think for now the comparison can be done manually, but a CI task to run diff on the previous CI tasks' output can be added.

[maflcko]

Another idea to limit the added files would be to re-create the merge step in the CI and abort if the difference in added files is larger than 30% or some magic value?

[murchandamus]

Maybe I’m just going about it wrong, but it’s kinda annoying to do it manually. So far I’ve been doing:

1. Reset to upstream/main
2. Run test_runner, check back a couple times at least thirty minutes later to grab the results
3. (Add and) fetch submitters remote repository, check out the PR branch
4. Run test_runner, check back a couple times at least thirty minutes later to grab the results
5. Run a few nifty search and replaces in vim to combine the two outputs such that each line is shown separately rather than all changes in block (I tried figuring out how to output the diff line-by-line with diff and git diff but didn’t find a satisfactory solution)

Am I overlooking an automated tool or smth?

[maflcko]

If you don't want to spend CPU locally, and you trust the CI, you can fetch the results from:

• https://cirrus-ci.com/github/bitcoin/bitcoin/master , for example https://cirrus-ci.com/task/5312073368862720?logs=ci#L4129
• The pull request here, for example https://cirrus-ci.com/task/5447597169573888?logs=ci#L4030

The last step will still have to be done manually as well.

[murchandamus]

I’ve been able to streamline the last step:

paste -d '\n' <(sed 's/^/-/' main-results-file) <(sed 's/^/+/' pr-results-file)