JUnit Reporter

[bitrunner]

I'm trying to make a JUnit reporter to get snitch test results into Jenkins. It may be useful for integrating with other existing tooling also.

The JUnit file format is XML with the root testsuites node containing a summary of all test results in attributes at the top of the file. This does not appear to fit the snitch reporter event driven model very well. One way to workaround this without caching the whole report in memory would be to output some kind of placeholder and then replace it with the actual summary data when the whole test run is finished. I don't see a convenient way to make that happen, so this might require some kind of change to the architecture.

JUnit also organizes tests into "suites" which doesn't map conveniently to a concept in snitch. A suite might sort of correspond to a tag but a test is part of one suite in JUnit, but can have multiple tags in snitch. One simple way to map them would be to just use all the tags associated with a test case as its suite. That would seem to require the whole result set to be cached and organized and then all output at the end. Alternatively, the tags could just be ignored and all tests could be put in a single suite.

I don't claim to comprehend the catch2 junit reporter, but it appears to cache all test results in memory and then create and write the whole XML tree when all tests are finished. It seems to also put all test cases under a single testsuite. Would that sort of design be acceptable in snitch?

If a JUnit report capability is desirable, let's hash out some kind of acceptable high level design.

[cschreib]

Hi and thanks for your interest!

Indeed the JUnit format is more troublesome to implement for the reason you mentioned (it does not work by streaming results).

One way to workaround this without caching the whole report in memory would be to output some kind of placeholder and then replace it with the actual summary data when the whole test run is finished. I don't see a convenient way to make that happen, so this might require some kind of change to the architecture.

A change to the architecture of snitch is unlikely at this point. However, I believe the event-driven / streaming reporting model is the lowest level implementation possible; although it's not an immediate good fit for JUnit, it should be possible to make it work. A custom JUnit reporter class can introduce additional abstraction or state to enable this, without touching the snitch internals. Getting this to work might be relatively expensive indeed (e.g., keeping track of all events until the end), and other reporting formats should not need to pay for it unless they also need it.

In Catch2 they can simply create and expand dynamic storage as needed to store all the test state. Here, we have some additional difficulties given snitch's additional constraint to avoid dynamic allocations. But there are several solutions possible:

1. The snitch tests already include code to do deep copies of events and store all of them for inspection at the end. This just requires a suitably large statically-allocated storage location (the size of which can be controlled at compile-time). We could reuse this. But it will introduce additional limitations that other reporting formats don't have (a maximum number of events, essentially).
2. If in-memory storage is too annoying, we could fall back to disk storage. Results could be streamed into temporary files, which are then read back to build the real report in a second pass. This won't work on platforms with no file system, though (indeed, snitch at present does not require a file system, except when explicitly requesting file output).
3. Perhaps this is closer to what you suggested: we could also build the final report in memory (as a very long string, essentially), write however much we can as we go along, leaving blank space for data not yet available, and go back to fill the gaps at the end. I would have to read the JUnit format more carefully; perhaps this could flush out most of the "expensive" event data (e.g., file names, line numbers, assertion messages), making the problem easier to solve.

Alternatively, the tags could just be ignored and all tests could be put in a single suite

This is what we currently do with the TeamCity reporter. A test suite is seen as a single test executable, essentially. I don't think tags are a suitable abstraction for representing a test suite; even though they can be used for that, they have other uses as well. Another model could be to map a snitch test case to a JUnit test suite, and a snitch section to a JUnit test case... But that might be confusing. I think it's probably best to stick to a single test suite for now.

I don't claim to comprehend the catch2 junit reporter, but it appears to cache all test results in memory and then create and write the whole XML tree when all tests are finished. It seems to also put all test cases under a single testsuite. Would that sort of design be acceptable in snitch?

Yes, I think so :) We claim compatibility with the Catch2 API, so following their behavior for reporters is a sensible route.