Truth seems to import JUnit 4, even on JUnit 5

[jbduncan]

It occurs to me that the core/ subproject's pom.xml imports JUnit 4 as a non-test dependency. I think this is fine for projects that use JUnit 4 and Truth together for testing, but I'm a little concerned that for projects using JUnit 5 it will cause JUnit 4 to be unnecessarily downloaded and depended upon.

Are there any plans to move any parts of Truth which depend on JUnit 4 APIs into their own extensions?

[cpovirk]

Unfortunately, the parts of Truth that depend on JUnit 4 are basically all "all of them," since we throw ComparisonFailure.

There are at least a couple approaches we could take here:

• Throw ComparisonFailure conditionally only if it's available at runtime.
• Adopt the opentest4j types once they're finalized.

Overall, I'm not too concerned about a dependency on JUnit 4 in Truth, since it's a test framework. But if it's causing people trouble, we should reconsider.

[dimo414]

Throwing ComparisonFailure only if it's already on the classpath would be a silent regression for users not using JUnit4, so (as you say, assuming it's not causing real trouble) I think waiting for opentest4j makes the most sense.

[kashike]

opentest4j 1.0.0 was released on September 10th.

[cpovirk]

Cool, thanks.

[Androbin]

Since #40 is merged, we can close this now.

[jbduncan]

@Androbin Sorry, I'm unclear on how #40 is related to this issue. Would you kindly clarify things for me?

[Androbin]

Oops, I meant cashapp/misk#40 😁

[cpovirk]

Oh, thanks for the link:

Until this is resolved, we should be careful to use the @org.junit.jupiter.api.Test annotation and not @org.junit.Test by mistake.

That's a good downside for us to keep in mind. Inside Google, I think we're pretty much resigned to having JUnit 4 on our classpath, and people haven't started using JUnit 5 much. But externally, things are different.

This is still something we could consider doing someday. It's just not at the top of the list currently.

[Androbin]

Isn't there a way to search & replace "en masse" across the codebase?
Or is this simply not feasible?

[cpovirk]

We can do search and replace. My fears are:

• Some people will be catching particular exception types, and their code will break. They ideally wouldn't be doing that, but we still have to clean them up inside Google, and others will have to fix it outside Google :\ Of course, the sooner we make the change, the better, so that the fallout is smaller.

• We'd need to get the opentest4j exceptions working under GWT and Android [update: all the way back to Ice Cream Sandwich] -- which is probably straightforward but sometimes turns out not to be.

• The opentest4j annotations still look possibly not officially finalized -- or maybe the docs just haven't been updated yet, since they're used in JUnit 5?

None of these are necessarily huge problems; there just didn't seem to be a big upside to eliminating the JUnit 4 dependency. The release of JUnit 5 may change that, and it calls attention to a larger problem with multiple @Test annotations.

[cpovirk]

Oh, and we should also check whether IDEs have begun supporting the opentest4j AssertionFailedError. I gather that they probably have, but I'd want to be sure. (e.g., I found this comment suggesting that IntelliJ has problems, but it seems hard to believe. I'd still want to verify.)

[childnode]

@Androbin

Isn't there a way to search & replace "en masse" across the codebase?
Or is this simply not feasible?

just have a look what the gradle guys do with their codebase: gradle/gradle#4116 << tl;dr: auto converting and running junit 4 tests with junit 5 platform both vintage and converted https://github.com/gradle/gradle/pull/4116/files#diff-a5945fda2451f7bab0e91c578e8e7f2eR49 ff

BUT: this is just done for the gradle tests itself and not intended to be used for other projects neither as production code

THIS truth one has other impacts, but if you ask me: just deprecate Junit 4 support and branch for Junit 5 ;)

[cpovirk]

We should at least consider migrating to the new extensions before 1.0. My guess is that we'll ultimately not consider this to be a true 1.0 blocker, but I'll mark it for consideration.

[cpovirk]

This might include eliminating TruthJUnit in favor of something on Truth itself.

Alternatively, it might include concluding that TruthJUnit is a weird name for a class that contains assume -- and then doing something about that.

[cpovirk]

Another thing that opentest4j offers is MultipleFailuresError, which would be nice for Expect.

[ptahchiev]

This issue is blocking me to migrate to JUnit5 :( I use google-compile testing framework which brings truth as transitive dependency and you guys have hard-coded junit4 classes in there (JUnitComparisonFailure)

[cpovirk]

My current thinking is that we won't fix this for 1.0 (which we're going to try to make our final release with breaking changes) because we can still theoretically fix it afterward by:

• changing our calls to FailureStrategy.fail to pass an AssertionFailedError (a subtype of AssertionError)
• making JUnit 4 an optional dependency that users can add if they want to use TruthJUnit
  I still hope to have a look at all that someday, but it's not imminent.

[cpovirk]

Oh, and I meant to add that, as part of opentest4j support, we'd also want to look for the opentest4j IncompleteExecutionException (and subtypes) in Expect.

[jbduncan]

Sounds good to me @cpovirk! 👍

[ptahchiev]

Any updates on this? I'd really like to see you guys support JUnit5 so we can move the google-compile-testing to JUnit5 too.

[cpovirk]

OK, 1.1.2 contains the fix (and another fix that I noticed I needed along the way...). Sorry for all the trouble.

It's still conceivable that something will go wrong in some case (InnerClasses?). But I have dealt with the one definite problem, which those following along at home may recognize as Java Puzzler 44 (as Martin Buchholz points out to me).

Please continue to let us know of new problems that the JUnit dependency causes. The ones I'm aware of:

• JUnit 5 users end up with 2 annotations named @Test on their classpath. This seems to be affecting a fair number of people. They may want to put tests/pre-commit hooks/static analysis in place to catch this error, or they may want to exclude our JUnit dependency at compile time (possible as of 1.1.2) -- and possibly also employ tbroyer's approach of a runtime-only dependency on JUnit 4, which lets Truth continue to throw a ComparisonFailure for better error reporting.
• IntelliJ tries to look for JUnit 4 tests. This week's report was the first I recall about that. As with the previous problem, those affected should be able to exclude the JUnit 4 dependency after upgrading to 1.1.2. They might also be able to employ tbroyer's workaround, but I'm unsure.

[KevinMcT]

1.1.2 looks good 👍 @cpovirk. No more broken window spam when utilizing truth in projects with jupiter and excluding junit.

(Work with stolsvik, so ack from us)

[MariusVolkhart]

@cpovirk the ExpectFailure class extends from JUnitTestRule which extends JUnit 4's TestRule. So when excluding JUnit 4, code referencing ExpectFailure no longer works. This includes code using the expectFailureAbout factory method

[tbroyer]

And that's, uh, expected: https://github.com/google/truth/releases/tag/release_1_1_2

(JUnit 4 is still required for some advanced features, like Expect, ExpectFailure, and TruthJUnit.assume().)

[MariusVolkhart]

🤦 missed that, thanks!

[cpovirk]

We don't have a clear plan for JUnit 5 at this point :(

I want to say that we ought to be able to extract the purely static version of ExpectFailure into a class with no JUnit dependency, but I'd have to confirm. And ideally, we'd want to get at least the beginnings of a long-term plan before copying things around.

In the meantime, copying and pasting the methods from ExpectFailure into your own codebase doesn't require too much duplication, so it may be a workable option (depending on licensing and other such questions).

[astubbs]

Related: Nicer way to support soft assertions (Expect.create junit4 Rule) in JUnit5? #893 (patch available)

[TWiStErRob]

@cpovirk is this still on hold?

[cpovirk]

Truth still declares a build dependency on JUnit:

truth/core/pom.xml

Lines 21 to 24 in ceecbd4

	<dependency>
	<groupId>junit</groupId>
	<artifactId>junit</artifactId>
	</dependency>

As of Truth 1.1.2, you can exclude that dependency, and Truth will work at runtime. (It just won't throw a ComparisonFailure.)

That's as far as we're likely to take this for a long while, unfortunately.

[TWiStErRob]

If that's the case would it make sense to document how to use Truth with JUnit 5 with detail on what features are lost to that exclusion and how to get them back in JUnit 5? I really like how detailed https://truth.dev/comparison is, would love to see a similar page for JUnit 5.

[cpovirk]

It would :( Truth is very much on the back burner. I do hope to change that someday.

Offhand, I think you lose only ComparisonFailure and the "assertThat alternatives," which are Expect, ExpectFailure, and TruthJUnit.assume() (noted in the 1.1.2 page). We don't have alternatives for them at the moment, nor do I know enough JUnit 5 to confidently sketch what they would look like (though I'd expect it to be relatively straightforward).

[Sineaggi]

According to the junit 5 codebase, it looks like they use the AssertionFailedError from org.opentest4j. Their asserts take the form of org.opentest4j.AssertionFailedError: expected: <1> but was: <2>