State of JDT LS tests with javac

[mickaelistria]

This issue is used to track some comments/progress ideas about making JDT-LS pass with Javac.
The reference job is https://ci.eclipse.org/ls/job/jdt-ls-javac/

Most tests unlisted here require fixes about bindings and problems mappings.

~10 tests failing because of Java 23:

• JVMConfiguratorTest.testPreviewFeatureSettings: (quoting @snjeza I can reproduce the issue in Eclipse with javac or ecj when starting the test with Java 23. https://github.com/eclipse-jdtls/eclipse.jdt.ls/blob/920a4bcc106f50b3a744d0104ce391b4cc1c7125/org.eclipse.jdt.ls.tests/src/org/eclipse/jdt/ls/core/internal/JVMConfiguratorTest.java#L184 returns 23. Java LS doesn't support Java 23 yet
• 5 "protobuf" test that so far fail complaining in the log with

* What went wrong:
BUG! exception in phase 'semantic analysis' in source unit '_BuildScript_' Unsupported class file major version 67
> Unsupported class file major version 67

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.
> Get more help at https://help.gradle.org.

CONFIGURE FAILED in 10s

It is an upstream buildship issue.
Buildship supports Gradle 8.9 - https://github.com/eclipse/buildship/blob/master/org.eclipse.buildship.core/src/main/resources/org/eclipse/buildship/core/internal/gradle/buildship-runtime.properties
Java 23 requires Gradle 8.10 - https://docs.gradle.org/current/userguide/compatibility.html
You can try the following patch:

 diff --git a/org.eclipse.jdt.ls.core/src/org/eclipse/jdt/ls/core/internal/managers/GradleProjectImporter.java b/org.eclipse.jdt.ls.core/src/org/eclipse/jdt/ls/core/internal/managers/GradleProjectImporter.java
index b8466681..0be495b4 100644
--- a/org.eclipse.jdt.ls.core/src/org/eclipse/jdt/ls/core/internal/managers/GradleProjectImporter.java
+++ b/org.eclipse.jdt.ls.core/src/org/eclipse/jdt/ls/core/internal/managers/GradleProjectImporter.java
@@ -102,7 +102,8 @@ public class GradleProjectImporter extends AbstractProjectImporter {
        public static final String SETTINGS_GRADLE_KTS_DESCRIPTOR = "settings.gradle.kts";
        public static final String GRADLE_WRAPPER_PROPERTIES_DESCRIPTOR = "gradle/wrapper/gradle-wrapper.properties";
 
-       public static final GradleDistribution DEFAULT_DISTRIBUTION = GradleDistribution.forVersion(GradleVersion.current().getVersion());
+       // public static final GradleDistribution DEFAULT_DISTRIBUTION = GradleDistribution.forVersion(GradleVersion.current().getVersion());
+       public static final GradleDistribution DEFAULT_DISTRIBUTION = GradleDistribution.forVersion("8.10");
 
        public static final String IMPORTING_GRADLE_PROJECTS = "Importing Gradle project(s)";

• 1 ProjectUtilsTest.testGetMaxProblemSeverity seem to fail for similar reason
• 4 GradleProjectImporterTest seem to fail for similar reason
• 1 issue about Lombok not being fully compatible with Java 23: [FEATURE] JDK 23 compatibility projectlombok/lombok#3722

~60 tests are about "missing features" of Javac over ECJ

• 28 "unused" or "deadcode" tests
• 1 ShowAllQuickFixTest.testShowAt() is also about missing marker for unused private fields
• 5 NullAnalysisTest since Javac doesn't provide null analysis
• 1 UnnecessaryCastQuickFixTest cannot work with Javac because Javac does not warn about useless cast
• 5 remaining JavadocQuickFixTest don't have javadoc -Xdoclint:all producing matching warning for 1. invalid qualifiers and 2. missing type parameters doc tag on top-level class
• 1 ConvertForLoopTest failing because of ConvertForLoopOperation cannot work with Javac #748
• 2 TypeMismatchQuickFixTest.testTypeMismatchInForEachMissingType() and UnresolvedTypesQuickfixText.testForEachMissingType() are failing because Javac cannot recover well from this paticular syntax error (not enough usable info to map the syntax error to JDT's typed error)
• 2 MissingEnumQuickFixTest cannot be working with plain Javac because javac doesn't warn/lint for such cases. Using switch expressions would enable warnings though.
• 3 DiagnosticHandlerTests that fail because of missing support for unused package, // TODO and dead code
• 3 remaining ModifiersCorrectionsQuickFixTests fail because Javac sees syntax errors in them and doesn't recover like ECJ. Fixing syntax errors usually make things move as expected.
• 1 CleanUpTest.testOverrideMethod is tracked in "Missing @Override" is not reported #770 and cannot easily be fixed without addition of extra static analysis.
• 2 ModifierCorrectionsQuickFixTest.testInvalid*Modifiers fail because Javac treat those as syntax errors and doesn't recover as expected by the quickfix, resulting in the quickfix not being shown.
• 1 RedundantInterfaceQuickFixTest.testRedundantSuperinterface failing because javac doesn't generate a warning for redundant interface
• 1 SerialVersionQuickFixTest.testAnonymousClass fails because Javac doesn't generate a warning for this case (but is a warning really meaningful here for a local class? ECJ might be too zealous in warning about it...)
• 1 ShowAllQuickFixTest.testDuplicateQuickFix2() has a syntax error that Javac cannot recover well enough to create quickfix proposal
• 1 TypeMismatchQuickfixText.testTypeMismatchWithTypeInSamePackage() is testing a case that is failing with ECJ because of package layout being more constrained, but which is valid with Javac (so no warning/error is emitted and thus no quickfix is provided)
• 1 UnresolvedTypeQuickFixTests.testParameterizedType2 Javac doesn't emit a 2nd TypeMismatch warning after the "unknown symbol" one, so no quickfix is provided here. But the expected quickfix to change to a type that is known to be unknown doesn't seem to be that interesting in a real usage scenario: maybe the newly created type will implement the target type...
• 1 UnresolvedVariablesQuickFixTest.testBug300() because Javac only report a syntax error on this statement and doesn't mention that the .z member doesn't exists.

Some tests don't seem to highlight a particular issue but instead should be modified to better accept Javac:

• 6 DocumentLifeCycleHandlerTest which seem to test specific JDT error messages, which are not the ones produced by Javac
• 2 DiagnosticsCommandTest fail because the message and their order are not the same ones as produced by ECJ (although they are equivalent)
• 1 ExtractVariableTest.testExtractVariable2 the edits we get with Javac is at least as good as what we get with ECJ (extracting 0 instead of 0/*]*/).
• 1 SignatureHelpHandlerTests seem to arbitrary expect one constructor (new String(byte[], Charset)) instead of an equivalently good proposal (new String(byte[], String)); but both are correct. Those failures do not hightlight a real bug or limitation, the test should be improved to work with either proposal.
• 4 GenerateDelegateMethodsHandlerTest which rely on the fact that JDT doesn't seem to recover private int B[] array; very well to pass, however Javac recovers it as B[] properly defined.
• 1 ReorgQuickFixTests.testWrongTypeName_bug180330 is tracked in
• 1 GetterSetterQuickFixTest.testInvisibleFieldToGetterSetter_2 fails because ECJ adds (useless) extra parenthesis. Javac behavior is fine, if not better. Not worth further work.
• 1 CompletionHandlerTest.testCompletion_Nullable() shows that we get a better (adds @Override and the caret location) proposal with Javac than with ECJ)
• 1 SemanticTokensHandlerTest.testSemanticTokens_Packages fails because Javac (rightly) recovered import java.NonExistentClass as a class name rather than a package name

Closed cases:

• 8 SignatureHelpHandlerTests requiring changes in JDT-LS: SignatureHelp activeSignature/activeParameter not set for DOM nodes with  eclipse.jdt.ls#3237
• Gradle tests that probably cannot work yet with Java 23 done on JDT-LS by @snjeza with eclipse-jdtls/eclipse.jdt.ls@0fc400f and eclipse-jdtls/eclipse.jdt.ls@2f37f45
• 3 "stream" tests showing some classloading error failed to access class org.eclipse.jdt.ls.core.internal.ConnectionStreamFactory$StreamProvider from class org.eclipse.jdt.ls.core.internal.ConnectionStreamFactoryTest (org.eclipse.jdt.ls.core.internal.ConnectionStreamFactory$StreamProvider is in unnamed module of loader org.eclipse.osgi.internal.loader.EquinoxClassLoader @5af468dd; org.eclipse.jdt.ls.core.internal.ConnectionStreamFactoryTest is in unnamed module of loader org.eclipse.osgi.internal.loader.EquinoxClassLoader @5f592ce4) fixed by @snjeza with Fix ConnectionStreamFactory with javac eclipse.jdt.ls#3258 ~
• Some AbstractMethodsQuickFixTest should be fixed with Allow "remove abstract" quickfix for BodyForAbstractMethod eclipse.jdt.ls#3259
• 2 (Prepare)RenameHandlerTests.testRenameTypeParameterInMethod() fails because Javac doesn't recover the same way as JDT in this case: the faulty code is treated as constructor rather than method. JDT-LS tests can be improved to more easily cover the initial case: Improve rename tests so they can work with Javac too eclipse.jdt.ls#3260

[snjeza]

All the 32 Gradle tests that probably cannot work yet with Java 23

@mickaelistria @rgrunber Could you try eclipse-jdtls/eclipse.jdt.ls#3238 ?