Restrict TestSpark action to suitable code types and fix generation for a line #344

arksap2002 · 2024-09-03T11:53:46Z

Description of changes made

The PR fixes the following issues:

Prohibits action use when non-Java and non-Kotlin file is opened.
Prohibits action use when the cursor is placed on a line that is surrounded by neither method nor class.
Successfully generates tests for a line outside of a method but within a class.
Fixes an issue with line numbering: use 1-based line number invariant for PsiHelper.getSurroundingLineNumber; Kotlin used 0-based numbering which attempted the generated prompt, namely it contained a line before the selected one.

See commit subjects and bodies for details.

What is left

The PR introduces a modification to core module (namely, extends the API of PromptGenerator with an overload of generatePromptForLine method that uses CUT as a line context). Therefore, I (@Vladislav0Art) need to publish a new version of the core module.

TODOs:

New core version 4.0.0 is published.

Other notes

Closes #336
Closes #352
Closes #247
Closes #249
Closes #377

I have checked that I am merging into the correct branch

pderakhshanfar · 2024-09-17T16:40:53Z

It does not solve #352 If I ask to generate test for package line in a kotlin project it throws the following exception:

java.lang.IllegalStateException
	at com.intellij.openapi.progress.util.AbstractProgressIndicatorBase.throwInvalidState(AbstractProgressIndicatorBase.java:130)
	at com.intellij.openapi.progress.util.AbstractProgressIndicatorBase.stop(AbstractProgressIndicatorBase.java:121)
	at com.intellij.openapi.progress.util.AbstractProgressIndicatorExBase.stop(AbstractProgressIndicatorExBase.java:52)
	at com.intellij.openapi.progress.util.AbstractProgressIndicatorExBase.delegate(AbstractProgressIndicatorExBase.java:211)
	at com.intellij.openapi.progress.util.AbstractProgressIndicatorExBase.doDelegateRunningChange(AbstractProgressIndicatorExBase.java:195)
	at com.intellij.openapi.progress.util.AbstractProgressIndicatorExBase.stop(AbstractProgressIndicatorExBase.java:53)
	at com.intellij.openapi.wm.impl.status.InfoAndProgressPanel$MyInlineProgressIndicator.stop(InfoAndProgressPanel.kt:830)
	at com.intellij.openapi.progress.util.AbstractProgressIndicatorExBase.delegate(AbstractProgressIndicatorExBase.java:211)
	at com.intellij.openapi.progress.util.AbstractProgressIndicatorExBase.doDelegateRunningChange(AbstractProgressIndicatorExBase.java:195)
	at com.intellij.openapi.progress.util.AbstractProgressIndicatorExBase.stop(AbstractProgressIndicatorExBase.java:53)
	at com.intellij.openapi.progress.util.ProgressWindow.stop(ProgressWindow.java:285)
	at org.jetbrains.research.testspark.display.custom.IJProgressIndicator.stop(IJProgressIndicator.kt:32)
	at org.jetbrains.research.testspark.tools.ToolUtils.isProcessCanceled(ToolUtils.kt:117)
	at org.jetbrains.research.testspark.tools.ToolUtils.isProcessStopped(ToolUtils.kt:112)
	at org.jetbrains.research.testspark.tools.Pipeline$runTestGeneration$1.run(Pipeline.kt:107)
	at com.intellij.openapi.progress.impl.CoreProgressManager.startTask(CoreProgressManager.java:477)
	at com.intellij.openapi.progress.impl.ProgressManagerImpl.startTask(ProgressManagerImpl.java:133)
	at com.intellij.openapi.progress.impl.CoreProgressManager.lambda$runProcessWithProgressAsynchronously$6(CoreProgressManager.java:528)
	at com.intellij.openapi.progress.impl.ProgressRunner.lambda$submit$4(ProgressRunner.java:250)
	at com.intellij.openapi.progress.ProgressManager.lambda$runProcess$0(ProgressManager.java:100)
	at com.intellij.openapi.progress.impl.CoreProgressManager.lambda$runProcess$1(CoreProgressManager.java:221)
	at com.intellij.platform.diagnostic.telemetry.helpers.TraceKt.use(trace.kt:46)
	at com.intellij.openapi.progress.impl.CoreProgressManager.lambda$runProcess$2(CoreProgressManager.java:220)
	at com.intellij.openapi.progress.impl.CoreProgressManager.lambda$executeProcessUnderProgress$13(CoreProgressManager.java:660)
	at com.intellij.openapi.progress.impl.CoreProgressManager.registerIndicatorAndRun(CoreProgressManager.java:735)
	at com.intellij.openapi.progress.impl.CoreProgressManager.computeUnderProgress(CoreProgressManager.java:691)
	at com.intellij.openapi.progress.impl.CoreProgressManager.executeProcessUnderProgress(CoreProgressManager.java:659)
	at com.intellij.openapi.progress.impl.ProgressManagerImpl.executeProcessUnderProgress(ProgressManagerImpl.java:79)
	at com.intellij.openapi.progress.impl.CoreProgressManager.runProcess(CoreProgressManager.java:202)
	at com.intellij.openapi.progress.ProgressManager.runProcess(ProgressManager.java:100)
	at com.intellij.openapi.progress.impl.ProgressRunner.lambda$submit$5(ProgressRunner.java:250)
	at com.intellij.openapi.progress.impl.ProgressRunner$ProgressRunnable.run(ProgressRunner.java:500)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
	at java.base/java.util.concurrent.Executors$PrivilegedThreadFactory$1$1.run(Executors.java:702)
	at java.base/java.util.concurrent.Executors$PrivilegedThreadFactory$1$1.run(Executors.java:699)
	at java.base/java.security.AccessController.doPrivileged(AccessController.java:399)
	at java.base/java.util.concurrent.Executors$PrivilegedThreadFactory$1.run(Executors.java:699)
	at java.base/java.lang.Thread.run(Thread.java:840)

java/src/main/kotlin/org/jetbrains/research/testspark/java/JavaPsiHelper.kt

Vladislav0Art · 2024-09-30T15:25:36Z

As @pderakhshanfar metioned above, if you attempt generation on an import you'l' get an exception. In my case, it was:

java.lang.NullPointerException
	at org.jetbrains.research.testspark.tools.llm.generation.PromptManager.getMethodDescriptor(PromptManager.kt:294)
	at org.jetbrains.research.testspark.tools.llm.generation.PromptManager.generatePrompt$lambda$6(PromptManager.kt:128)
	at com.intellij.openapi.application.impl.RwLockHolder.runReadAction(RwLockHolder.kt:271)
	at com.intellij.openapi.application.impl.ApplicationImpl.runReadAction(ApplicationImpl.java:845)
	at org.jetbrains.research.testspark.tools.llm.generation.PromptManager.generatePrompt(PromptManager.kt:69)
	at org.jetbrains.research.testspark.tools.llm.generation.LLMProcessManager.runTestGenerator(LLMProcessManager.kt:105)
	at org.jetbrains.research.testspark.tools.Pipeline$runTestGeneration$1.run(Pipeline.kt:96)
	at com.intellij.openapi.progress.impl.CoreProgressManager.startTask(CoreProgressManager.java:477)
	at com.intellij.openapi.progress.impl.ProgressManagerImpl.startTask(ProgressManagerImpl.java:133)
	at com.intellij.openapi.progress.impl.CoreProgressManager.lambda$runProcessWithProgressAsynchronously$6(CoreProgressManager.java:528)
	at com.intellij.openapi.progress.impl.ProgressRunner.lambda$submit$4(ProgressRunner.java:250)
	at com.intellij.openapi.progress.ProgressManager.lambda$runProcess$0(ProgressManager.java:100)
	at com.intellij.openapi.progress.impl.CoreProgressManager.lambda$runProcess$1(CoreProgressManager.java:221)
	at com.intellij.platform.diagnostic.telemetry.helpers.TraceKt.use(trace.kt:46)
	at com.intellij.openapi.progress.impl.CoreProgressManager.lambda$runProcess$2(CoreProgressManager.java:220)
	at com.intellij.openapi.progress.impl.CoreProgressManager.lambda$executeProcessUnderProgress$13(CoreProgressManager.java:660)
	at com.intellij.openapi.progress.impl.CoreProgressManager.registerIndicatorAndRun(CoreProgressManager.java:735)
	at com.intellij.openapi.progress.impl.CoreProgressManager.computeUnderProgress(CoreProgressManager.java:691)
	at com.intellij.openapi.progress.impl.CoreProgressManager.executeProcessUnderProgress(CoreProgressManager.java:659)
	at com.intellij.openapi.progress.impl.ProgressManagerImpl.executeProcessUnderProgress(ProgressManagerImpl.java:79)
	at com.intellij.openapi.progress.impl.CoreProgressManager.runProcess(CoreProgressManager.java:202)
	at com.intellij.openapi.progress.ProgressManager.runProcess(ProgressManager.java:100)
	at com.intellij.openapi.progress.impl.ProgressRunner.lambda$submit$5(ProgressRunner.java:250)
	at com.intellij.openapi.progress.impl.ProgressRunner$ProgressRunnable.run(ProgressRunner.java:500)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
	at java.base/java.util.concurrent.Executors$PrivilegedThreadFactory$1$1.run(Executors.java:702)
	at java.base/java.util.concurrent.Executors$PrivilegedThreadFactory$1$1.run(Executors.java:699)
	at java.base/java.security.AccessController.doPrivileged(AccessController.java:399)
	at java.base/java.util.concurrent.Executors$PrivilegedThreadFactory$1.run(Executors.java:699)
	at java.base/java.lang.Thread.run(Thread.java:840)
2024-09-30 17:18:58,507 [  38600] SEVERE - #c.i.o.p.Task - IntelliJ IDEA 2024.1  Build #IC-241.14494.240
2024-09-30 17:18:58,507 [  38600] SEVERE - #c.i.o.p.Task - JDK: 17.0.10; VM: OpenJDK 64-Bit Server VM; Vendor: JetBrains s.r.o.
2024-09-30 17:18:58,507 [  38600] SEVERE - #c.i.o.p.Task - OS: Mac OS X
2024-09-30 17:18:58,508 [  38601] SEVERE - #c.i.o.p.Task - Plugin to blame: TestSpark version: 0.2.1
2024-09-30 17:18:58,508 [  38601] SEVERE - #c.i.o.p.Task - Last Action: TestSpark.TestSparkActions
2024-09-30 17:18:58,510 [  38603] SEVERE - #c.i.o.a.i.FlushQueue - null

It is because we assert that a surrounding method exists. In general, we should quit using !! in TestSpark. It has always dealt us a bad hand:

We should hide the action if there is no valid code construct under test. For a given line, we need to ensure it is within a method, class, or top-level function. The check should be performed in this order: 1) search for a surrounding method; if not found, 2) check for a surrounding class; if not found, 3) check for a top-level function. If none exist, the line is top-level, and we cannot generate tests, so the action should not appear in the right-click menu.

Vladislav0Art · 2024-09-30T15:38:08Z

The same error as here occurs if I start generation for a line (**) and select line as a code construct under test:

public class Calc {
    int a = 10; // (**)

    public int sum(int a, int b) {
        return a + b;
    }

    public int multiply(int a, int b) {
        return a * b;
    }
}

Vladislav0Art

There are important comments above that identify unfixed bugs. Those should be addressed in this PR to close the issue #352.

java/src/main/kotlin/org/jetbrains/research/testspark/java/JavaPsiHelper.kt

langwrappers/src/main/kotlin/org/jetbrains/research/testspark/langwrappers/PsiComponents.kt

src/main/kotlin/org/jetbrains/research/testspark/actions/TestSparkAction.kt

Vladislav0Art · 2024-10-04T14:46:07Z

It does not solve #352 If I ask to generate test for package line in a kotlin project it throws the following exception:

Fixed.

Vladislav0Art · 2024-10-07T12:48:42Z

The same error as here occurs if I start generation for a line (**) and select line as a code construct under test:

Fixed.

Vladislav0Art · 2024-10-07T12:49:39Z

As @pderakhshanfar metioned above, if you attempt generation on an import you'l' get an exception. In my case, it was:

Fixed.

Vladislav0Art

Introduced modifications that solve the described problems.

Vladislav0Art

I must approve the PR to allow its further merge. This approval does not count.

Hello-zoka · 2024-10-07T20:11:04Z

I've found several problems connected with line generation:

It doesn't include information about the class line refers to. For example in method generation we have smth like: Generate unit tests in kotlin for org.example.Calc.mul to achieve 100% line coverage for this method.
For example, when asked to generate a test for the line 'return a + b' in the following code:

class Calc {
    fun sum(a: Int, b: Int): Int {
        return a + b
    }
 }

LLM is struggling with generating Calc().sum() calls and spending attempts on generating uncompilable sum() calls. This leads to unnecessary feedback cycles iterations.

This problem became even more crucial if attempted to generate a test for a line inside a companion object, or it's method. LLM just don't know anything about how to call such function.

Solution: includes information about the class surrounding the line if such is present.

We are not allowing line generation for empty lines, but it can be asked for comment lines if it is surrounded by class/function. The result will be almost the same as for the whole class test generation, but it's weird
Adding to the previous point, there is a strange restriction for such a top-level function:

/// 123 -- not allowed for line generation

// 123 -- allowed for line generation
fun multwice(a: Int) = a * 2

Solution idea: For each line generation, check if it includes something besides comments, open/closed brackets and allow generation only in such case

Vladislav0Art · 2024-10-08T07:57:15Z

I've found several problems connected with line generation:

It doesn't include information about the class line refers to. For example in method generation we have smth like: Generate unit tests in kotlin for org.example.Calc.mul to achieve 100% line coverage for this method.
For example, when asked to generate a test for the line 'return a + b' in the following code:
class Calc {
    fun sum(a: Int, b: Int): Int {
        return a + b
    }
 }
LLM is struggling with generating Calc().sum() calls and spending attempts on generating uncompilable sum() calls. This leads to unnecessary feedback cycles iterations.

This problem became even more crucial if attempted to generate a test for a line inside a companion object, or it's method. LLM just don't know anything about how to call such function.

Solution: includes information about the class surrounding the line if such is present.

We are not allowing line generation for empty lines, but it can be asked for comment lines if it is surrounded by class/function. The result will be almost the same as for the whole class test generation, but it's weird

Adding to the previous point, there is a strange restriction for such a top-level function:
/// 123 -- not allowed for line generation

// 123 -- allowed for line generation
fun multwice(a: Int) = a * 2
Solution idea: For each line generation, check if it includes something besides comments, open/closed brackets and allow generation only in such case

@Hello-zoka, I agree with the 2nd and 3rd points; I'll address them in the PR or explain why they should be addressed in a different PR.

At first glance, the first point does not appear to be an easy task, and it does not represent a bug; it is a lack of context. It is a stand-alone issue that should be addressed in a different PR (possibly after the upcoming release). @Hello-zoka, could you create an issue for it?

My main focus, for now, is to close as many issues as possible that yield runtime exceptions and fail the plugin. The aspects regarding line-based test generation mentioned appear minor to me as very few people use line-based test generation compared to method/class-based generation (a good thing to calculate it as a metric! @arksap2002).

CC: @pderakhshanfar

Vladislav0Art · 2024-10-08T10:20:54Z

I've found several problems connected with line generation:

It doesn't include information about the class line refers to. For example in method generation we have smth like: Generate unit tests in kotlin for org.example.Calc.mul to achieve 100% line coverage for this method.
For example, when asked to generate a test for the line 'return a + b' in the following code:
class Calc {
    fun sum(a: Int, b: Int): Int {
        return a + b
    }
 }
LLM is struggling with generating Calc().sum() calls and spending attempts on generating uncompilable sum() calls. This leads to unnecessary feedback cycles iterations.

This problem became even more crucial if attempted to generate a test for a line inside a companion object, or it's method. LLM just don't know anything about how to call such function.

Solution: includes information about the class surrounding the line if such is present.

We are not allowing line generation for empty lines, but it can be asked for comment lines if it is surrounded by class/function. The result will be almost the same as for the whole class test generation, but it's weird

Adding to the previous point, there is a strange restriction for such a top-level function:
/// 123 -- not allowed for line generation

// 123 -- allowed for line generation
fun multwice(a: Int) = a * 2
Solution idea: For each line generation, check if it includes something besides comments, open/closed brackets and allow generation only in such case

@Hello-zoka
Regarding points 2 and 3, I will create a separate issue for them. Both require modifications in how we select PSI elements in PsiHelper.getCurrentListOfCodeTypes + it needs additional parsing, which is out of the scope of the current PR.

Reflected in #377, #378.

Hello-zoka

It seems like it is easy to fix #377 by refactoring how you generate the prompt for the line. In my opinion, it has to be in this PR since it is about "fix generation for the line" + #337 depends on the changes from this PR.

...c/main/kotlin/org/jetbrains/research/testspark/core/generation/llm/prompt/PromptGenerator.kt

src/main/kotlin/org/jetbrains/research/testspark/actions/TestSparkAction.kt

src/main/kotlin/org/jetbrains/research/testspark/tools/llm/generation/PromptManager.kt

pderakhshanfar

Nit (and we dont need to resolve it in this issue) TestSpark option is still available when we try to generate test for a test file:
open a test file right click and ask for test generation using TestSpark

…orGeneration`

The bug is reflected in the issue #375.

…umbers Before, the `KotlinPsiHelper` returned a 0-based line number which caused an issue with line-based test generation. The generated prompt contained a line above the selected one.

When there is no surrounding method about the selected line, we use the CUT as a context for this line. The CUT must always be present. Otherwise, the generation action should have been disabled for this line.

The line-based test generation that has a method as a context of the line now also accepts constructors of the containing class.

The major version increased due to the change of the public API of `PromptGenerator.generatePromptForLine` method.

…or a line (JetBrains-Research#344) * fix update function * create availableForGeneration * ktlint * feat: add javadoc for `JavaPsiHelper.availableForGeneration` * feat: check for nullness of a PSI file in `TestSparkAction.update` * feat: update javadocs in `PsiComponents.kt` * feat: check for a class or method/func in `KotlinPsiHelper.availableForGeneration` * feat: add TODO to `ToolUtils` about a potential bug The bug is reflected in the issue JetBrains-Research#375. * feat: make `PsiHelper.getSurroundingLineNumber` return 1-based line numbers Before, the `KotlinPsiHelper` returned a 0-based line number which caused an issue with line-based test generation. The generated prompt contained a line above the selected one. * feat: implement line-based test generation with CUT as a context When there is no surrounding method about the selected line, we use the CUT as a context for this line. The CUT must always be present. Otherwise, the generation action should have been disabled for this line. * refactor: apply ktlint * feat: add `See` in TODO * feat: add TODO and surround $NAME in backticks in `linePrompt` template * feat: collect class constructor signatures in `PsiClassWrapper` * feat: remove backticks from `linePrompt` * feat: fill line-based test generation with additional context The line-based test generation that has a method as a context of the line now also accepts constructors of the containing class. * refactor: use `firstOrNull` for `cut` extraction * refactor: apply ktlint * fix: add required parameter to `ClassRepresentation` in tests * publish: core module version `4.0.0` The major version increased due to the change of the public API of `PromptGenerator.generatePromptForLine` method. --------- Co-authored-by: Vladislav Artiukhov <[email protected]>

@after

* minimal attempt to run kex from testspark with harcoded settings Needs debugging and testing * load kex-runner jar from github (build.gradle.kts toplevel) * setup code for kex properties * KexErrorManager based on LLMErrorManager * KexProcessManager based on EvoSuiteProcessManager * Basic UI element (button for running kex) * kex works only for the class codeType (todo funciton and line if possible) * read resource files kex.policy and modules.info * generate Report and series of simplifications for MVP use a provided kex path for now instead of downloading jar and adding dependency use ProcessBuilder (jdk) instead of OSProcessHandler (IJ sdk) use kex.py instead of building java command directly generate Report objects by reading generated java classes * download kex as github release if it doesn't exist * delete commented code Deleted stuff: running Kex through through OSProcessHandler (IJ sdk) running Kex with kex.py downloading kex from github in build.gradle.kts * use javaparser to merge @before and @after generated methods * compiling tests successfully by adding helper method to TestGenData.otherInfo * fold anything but @test annotated code, but unfortunately only on UI updates * remove python dependency by running kex jar directly * use project's java and add version check * empty implementation of settings classes for kex * allow setting kex arguments from kex settings page * kex test generation for 'method' code type arguably overkill solution of modify PsiMethodWrapper class added parameter names, types and return tyepes explicitly updated implementation for java and kotlin * fold all but @test methods regardless of declaration order * remove settings for unsupported kex features * folding works even if junit isn't in classpath * use the the subprocess manager from IJ framework * empty kex path download to .cache and LOCALAPPDATA in windows * fix lint. remove wildcard imports * from IJ api, provide correct build directory to kex For multimodule projects using TestSpark the correct module's build directory path is passed based on the location of code for which tests are generated * set maxTests displayed and minimizeTestSuite options * fix code fold to only happen once at the start not every ui update * error handling for errors in kex process manager also fixed a bug with the way options were passed to kex subprocess made them a list of strings instead of a single big space separated string * fix lint * provide signatures with FQNs to kex for java This also includes a simple string based mapping to jvm types through type erasure * refactor. extract functons, add comments, remove redundancies * fix lint * every option in kex cmd is preceeded by --option * use kotlin.time.Duration instead of Int * refactor error handling code, check for non-zero exit code * make generated code manipulation more robust It no longer depends on the order of the methods generated * only check if the correct verison of kex exists * make download url a property and version a user setting * support jdk version 8. no --add-opens * bump up kex version to 0.0.10 * allow kex test generation for classes not in a package * null safety while folding helper code * Merge branch 'development' into edwin1729/improvement/kex-integration * disallow kex and line code type being chosen simultaneously * update description tab and README.md with kex * fix lint * fix run coverage by making method name same as class name This is expected by TestSpark. Relevant file TestProcess.kt:createXmlFromJacoco:109 * undo mistaken removal of addLanguageTextFieldListener before code folding * Add empty LLM response handling (#342) Show a warning if LLM returns a response that cannot be parsed into a test case (e.g., an explanation of this test case rather than a modification). * Restrict TestSpark action to suitable code types and fix generation for a line (#344) * fix update function * create availableForGeneration * ktlint * feat: add javadoc for `JavaPsiHelper.availableForGeneration` * feat: check for nullness of a PSI file in `TestSparkAction.update` * feat: update javadocs in `PsiComponents.kt` * feat: check for a class or method/func in `KotlinPsiHelper.availableForGeneration` * feat: add TODO to `ToolUtils` about a potential bug The bug is reflected in the issue #375. * feat: make `PsiHelper.getSurroundingLineNumber` return 1-based line numbers Before, the `KotlinPsiHelper` returned a 0-based line number which caused an issue with line-based test generation. The generated prompt contained a line above the selected one. * feat: implement line-based test generation with CUT as a context When there is no surrounding method about the selected line, we use the CUT as a context for this line. The CUT must always be present. Otherwise, the generation action should have been disabled for this line. * refactor: apply ktlint * feat: add `See` in TODO * feat: add TODO and surround $NAME in backticks in `linePrompt` template * feat: collect class constructor signatures in `PsiClassWrapper` * feat: remove backticks from `linePrompt` * feat: fill line-based test generation with additional context The line-based test generation that has a method as a context of the line now also accepts constructors of the containing class. * refactor: use `firstOrNull` for `cut` extraction * refactor: apply ktlint * fix: add required parameter to `ClassRepresentation` in tests * publish: core module version `4.0.0` The major version increased due to the change of the public API of `PromptGenerator.generatePromptForLine` method. --------- Co-authored-by: Vladislav Artiukhov <[email protected]> * Data class for execution results * Minor refactoring * apply ktlint * fix getJavaVersion * add KexSettingsService to plugin.xml * fix DefaultKexSettingsState * fix KexSettingsState * fix Bundles * Force junit4 for EvoSuite tests * JUnit version forcing * fix getExceptionData * fix tools * fix managers * fix TestCasePanelBuilder.kt * fix TestProcessor.kt * remove unused import * fix ktlint * fix ktlint * remove TestSparkStarter.kt * Add support for language applicability checks in Tool * Add support for language applicability checks in Llm * Add support for language applicability checks in Kex * Add support for language applicability checks in EvoSuite * Filter test generation buttons by language compatibility * Refactor imports for SupportedLanguage in EvoSuite and Llm * Set progress indicator text during Kex test generation --------- Co-authored-by: Edwin Fernando <[email protected]> Co-authored-by: Edwin Fernando <[email protected]> Co-authored-by: Edwin Fernando <[email protected]> Co-authored-by: Vladislav Artiukhov <[email protected]> Co-authored-by: Iurii Zaitsev <[email protected]> Co-authored-by: Hello-zoka <[email protected]>

@after

* minimal attempt to run kex from testspark with harcoded settings Needs debugging and testing * load kex-runner jar from github (build.gradle.kts toplevel) * setup code for kex properties * KexErrorManager based on LLMErrorManager * KexProcessManager based on EvoSuiteProcessManager * Basic UI element (button for running kex) * kex works only for the class codeType (todo funciton and line if possible) * read resource files kex.policy and modules.info * generate Report and series of simplifications for MVP use a provided kex path for now instead of downloading jar and adding dependency use ProcessBuilder (jdk) instead of OSProcessHandler (IJ sdk) use kex.py instead of building java command directly generate Report objects by reading generated java classes * download kex as github release if it doesn't exist * delete commented code Deleted stuff: running Kex through through OSProcessHandler (IJ sdk) running Kex with kex.py downloading kex from github in build.gradle.kts * use javaparser to merge @before and @after generated methods * compiling tests successfully by adding helper method to TestGenData.otherInfo * fold anything but @test annotated code, but unfortunately only on UI updates * remove python dependency by running kex jar directly * use project's java and add version check * empty implementation of settings classes for kex * allow setting kex arguments from kex settings page * kex test generation for 'method' code type arguably overkill solution of modify PsiMethodWrapper class added parameter names, types and return tyepes explicitly updated implementation for java and kotlin * fold all but @test methods regardless of declaration order * remove settings for unsupported kex features * folding works even if junit isn't in classpath * use the the subprocess manager from IJ framework * empty kex path download to .cache and LOCALAPPDATA in windows * fix lint. remove wildcard imports * from IJ api, provide correct build directory to kex For multimodule projects using TestSpark the correct module's build directory path is passed based on the location of code for which tests are generated * set maxTests displayed and minimizeTestSuite options * fix code fold to only happen once at the start not every ui update * error handling for errors in kex process manager also fixed a bug with the way options were passed to kex subprocess made them a list of strings instead of a single big space separated string * fix lint * provide signatures with FQNs to kex for java This also includes a simple string based mapping to jvm types through type erasure * refactor. extract functons, add comments, remove redundancies * fix lint * every option in kex cmd is preceeded by --option * use kotlin.time.Duration instead of Int * refactor error handling code, check for non-zero exit code * make generated code manipulation more robust It no longer depends on the order of the methods generated * only check if the correct verison of kex exists * make download url a property and version a user setting * support jdk version 8. no --add-opens * bump up kex version to 0.0.10 * allow kex test generation for classes not in a package * null safety while folding helper code * Merge branch 'development' into edwin1729/improvement/kex-integration * disallow kex and line code type being chosen simultaneously * update description tab and README.md with kex * fix lint * fix run coverage by making method name same as class name This is expected by TestSpark. Relevant file TestProcess.kt:createXmlFromJacoco:109 * undo mistaken removal of addLanguageTextFieldListener before code folding * Add empty LLM response handling (#342) Show a warning if LLM returns a response that cannot be parsed into a test case (e.g., an explanation of this test case rather than a modification). * Restrict TestSpark action to suitable code types and fix generation for a line (#344) * fix update function * create availableForGeneration * ktlint * feat: add javadoc for `JavaPsiHelper.availableForGeneration` * feat: check for nullness of a PSI file in `TestSparkAction.update` * feat: update javadocs in `PsiComponents.kt` * feat: check for a class or method/func in `KotlinPsiHelper.availableForGeneration` * feat: add TODO to `ToolUtils` about a potential bug The bug is reflected in the issue #375. * feat: make `PsiHelper.getSurroundingLineNumber` return 1-based line numbers Before, the `KotlinPsiHelper` returned a 0-based line number which caused an issue with line-based test generation. The generated prompt contained a line above the selected one. * feat: implement line-based test generation with CUT as a context When there is no surrounding method about the selected line, we use the CUT as a context for this line. The CUT must always be present. Otherwise, the generation action should have been disabled for this line. * refactor: apply ktlint * feat: add `See` in TODO * feat: add TODO and surround $NAME in backticks in `linePrompt` template * feat: collect class constructor signatures in `PsiClassWrapper` * feat: remove backticks from `linePrompt` * feat: fill line-based test generation with additional context The line-based test generation that has a method as a context of the line now also accepts constructors of the containing class. * refactor: use `firstOrNull` for `cut` extraction * refactor: apply ktlint * fix: add required parameter to `ClassRepresentation` in tests * publish: core module version `4.0.0` The major version increased due to the change of the public API of `PromptGenerator.generatePromptForLine` method. --------- Co-authored-by: Vladislav Artiukhov <[email protected]> * Data class for execution results * Minor refactoring * apply ktlint * fix getJavaVersion * add KexSettingsService to plugin.xml * fix DefaultKexSettingsState * fix KexSettingsState * fix Bundles * Force junit4 for EvoSuite tests * JUnit version forcing * fix getExceptionData * fix tools * fix managers * fix TestCasePanelBuilder.kt * fix TestProcessor.kt * remove unused import * fix ktlint * fix ktlint * remove TestSparkStarter.kt * Add support for language applicability checks in Tool * Add support for language applicability checks in Llm * Add support for language applicability checks in Kex * Add support for language applicability checks in EvoSuite * Filter test generation buttons by language compatibility * Refactor imports for SupportedLanguage in EvoSuite and Llm * Set progress indicator text during Kex test generation --------- Co-authored-by: Edwin Fernando <[email protected]> Co-authored-by: Edwin Fernando <[email protected]> Co-authored-by: Edwin Fernando <[email protected]> Co-authored-by: Vladislav Artiukhov <[email protected]> Co-authored-by: Iurii Zaitsev <[email protected]> Co-authored-by: Hello-zoka <[email protected]>

arksap2002 added bug Something isn't working Ready for review labels Sep 3, 2024

pderakhshanfar self-requested a review September 17, 2024 16:41

pderakhshanfar added the In progress label Sep 17, 2024

Hello-zoka reviewed Sep 27, 2024

View reviewed changes

java/src/main/kotlin/org/jetbrains/research/testspark/java/JavaPsiHelper.kt Outdated Show resolved Hide resolved

Vladislav0Art requested changes Sep 30, 2024

View reviewed changes

Vladislav0Art removed the Ready for review label Oct 4, 2024

Vladislav0Art self-assigned this Oct 4, 2024

Vladislav0Art force-pushed the arksap2002/bugs/fix-anaction-update branch from b40a077 to 3079173 Compare October 4, 2024 14:39

Vladislav0Art changed the title ~~Fix TestSparkAction/update function~~ Restrict TestSpark action to suitable code types and fix generation for a line Oct 7, 2024

Vladislav0Art mentioned this pull request Oct 7, 2024

Fix line prompt managment #356

Closed

1 task

Vladislav0Art self-requested a review October 7, 2024 12:50

Vladislav0Art reviewed Oct 7, 2024

View reviewed changes

Vladislav0Art self-requested a review October 7, 2024 12:50

Vladislav0Art approved these changes Oct 7, 2024

View reviewed changes

Vladislav0Art added Ready for review and removed In progress labels Oct 7, 2024

Vladislav0Art requested a review from Hello-zoka October 7, 2024 13:01

Vladislav0Art added the urgent Urgent PR label Oct 7, 2024

Vladislav0Art mentioned this pull request Oct 8, 2024

Insufficient context about the surrounding method for line-based test generation #377

Closed

Hello-zoka requested changes Oct 8, 2024

View reviewed changes

Hello-zoka approved these changes Oct 9, 2024

View reviewed changes

pderakhshanfar approved these changes Oct 10, 2024

View reviewed changes

arksap2002 and others added 19 commits October 10, 2024 12:25

fix update function

dd2b904

create availableForGeneration

9a915a1

ktlint

ef63467

feat: add javadoc for JavaPsiHelper.availableForGeneration

90e43d6

feat: check for nullness of a PSI file in TestSparkAction.update

3a907b0

feat: update javadocs in PsiComponents.kt

bb473bc

feat: check for a class or method/func in `KotlinPsiHelper.availableF…

f682df3

…orGeneration`

feat: add TODO to ToolUtils about a potential bug

c8a61e6

The bug is reflected in the issue #375.

feat: make PsiHelper.getSurroundingLineNumber return 1-based line n…

52d9c5d

…umbers Before, the `KotlinPsiHelper` returned a 0-based line number which caused an issue with line-based test generation. The generated prompt contained a line above the selected one.

feat: implement line-based test generation with CUT as a context

817652c

When there is no surrounding method about the selected line, we use the CUT as a context for this line. The CUT must always be present. Otherwise, the generation action should have been disabled for this line.

refactor: apply ktlint

d657a7b

feat: add See in TODO

f12434e

feat: add TODO and surround $NAME in backticks in linePrompt template

2338c4d

feat: collect class constructor signatures in PsiClassWrapper

a62954e

feat: remove backticks from linePrompt

22b6319

feat: fill line-based test generation with additional context

72163f4

The line-based test generation that has a method as a context of the line now also accepts constructors of the containing class.

refactor: use firstOrNull for cut extraction

46e1cae

refactor: apply ktlint

8caadb0

fix: add required parameter to ClassRepresentation in tests

7d99e58

Vladislav0Art force-pushed the arksap2002/bugs/fix-anaction-update branch from eb19a43 to 7d99e58 Compare October 10, 2024 10:30

publish: core module version 4.0.0

abb89cc

The major version increased due to the change of the public API of `PromptGenerator.generatePromptForLine` method.

Vladislav0Art merged commit d696259 into development Oct 10, 2024
3 checks passed

arksap2002 deleted the arksap2002/bugs/fix-anaction-update branch October 15, 2024 21:12

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Restrict TestSpark action to suitable code types and fix generation for a line #344

Restrict TestSpark action to suitable code types and fix generation for a line #344

arksap2002 commented Sep 3, 2024 •

edited by Vladislav0Art

Loading

pderakhshanfar commented Sep 17, 2024

Vladislav0Art commented Sep 30, 2024 •

edited

Loading

Vladislav0Art commented Sep 30, 2024

Vladislav0Art left a comment

Vladislav0Art commented Oct 4, 2024

Vladislav0Art commented Oct 7, 2024

Vladislav0Art commented Oct 7, 2024

Vladislav0Art left a comment

Vladislav0Art left a comment

Hello-zoka commented Oct 7, 2024 •

edited

Loading

Vladislav0Art commented Oct 8, 2024

Vladislav0Art commented Oct 8, 2024 •

edited

Loading

Hello-zoka left a comment •

edited

Loading

pderakhshanfar left a comment

Restrict TestSpark action to suitable code types and fix generation for a line #344

Restrict TestSpark action to suitable code types and fix generation for a line #344

Conversation

arksap2002 commented Sep 3, 2024 • edited by Vladislav0Art Loading

Description of changes made

What is left

Other notes

pderakhshanfar commented Sep 17, 2024

Vladislav0Art commented Sep 30, 2024 • edited Loading

Vladislav0Art commented Sep 30, 2024

Vladislav0Art left a comment

Choose a reason for hiding this comment

Vladislav0Art commented Oct 4, 2024

Vladislav0Art commented Oct 7, 2024

Vladislav0Art commented Oct 7, 2024

Vladislav0Art left a comment

Choose a reason for hiding this comment

Vladislav0Art left a comment

Choose a reason for hiding this comment

Hello-zoka commented Oct 7, 2024 • edited Loading

Vladislav0Art commented Oct 8, 2024

Vladislav0Art commented Oct 8, 2024 • edited Loading

Hello-zoka left a comment • edited Loading

Choose a reason for hiding this comment

pderakhshanfar left a comment

Choose a reason for hiding this comment

arksap2002 commented Sep 3, 2024 •

edited by Vladislav0Art

Loading

Vladislav0Art commented Sep 30, 2024 •

edited

Loading

Hello-zoka commented Oct 7, 2024 •

edited

Loading

Vladislav0Art commented Oct 8, 2024 •

edited

Loading

Hello-zoka left a comment •

edited

Loading