Should results merging happen in CI?

[gsnedders]

Currently, the results processor receives either: a list of wptreport JSON URLs and a list of screenshot URLs, or a single ZIP URL, where the ZIP contains a number of wptreport JSON files and a number of screenshot files.

The results processor then attempts to merge all these files together.

We currently have a bunch of things we ignore when merging all the files together:

wpt.fyi/results-processor/wptreport.py

Lines 38 to 41 in 5682248

	# Ignore inconsistent browser minor versions for now.
	# TODO(Hexcles): Remove this when the TC decision task is implemented.
	IGNORED_CONFLICTS = frozenset({'browser_build_id', 'browser_changeset',
	'version', 'os_build'})

Ideally we'd put the logic somewhere closer to CI where we can actually ignore fields that make sense for that specific configuration.

GitHub Actions also always compresses artifacts, and if we're using multiple jobs for parallelism (the only sensible approach!), then we end up with multiple ZIP files which need to get merged (which we don't currently support).

[past]

I'm OK with this in principle, but it means that we have to make this change across all CIs, right?

[gsnedders]

@past Right. I'd imagine we create some wpt merge-results or something where the logic actually lives, but obviously this involves more intermediate artifacts being created in every CI system.

cc @jgraham

[jgraham]

I'm basically happy with the idea of CI being taught to merge the results of multiple runs into one giant JSON file; having all the detailed results from a run in one place can sometimes be useful independent of wpt.fyi, so I think that is enough to make this worth doing.