Skip to content

Commit

Permalink
Changed name module and minimal improvement. (#3)
Browse files Browse the repository at this point in the history
- Changed name module from retorch-llm-rp to llm-rp-expstudy
- Improved maintainability of the Helper class including one global variable with the route to the different resources
  • Loading branch information
augustocristian authored Sep 20, 2024
1 parent f83c8e6 commit a27e2a5
Show file tree
Hide file tree
Showing 17 changed files with 19 additions and 18 deletions.
14 changes: 7 additions & 7 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,10 +16,10 @@ The replication package is structured as follows:

1. `/docs`: contains the experimental outputs as well as the experimental baselines, namely as aforementioned .

2. `/retorch-llm-rp/src/main`: contains all the necessary Java scripting code for execute the different prompts to the
2. `/llm-rp-expstudy/src/main`: contains all the necessary Java scripting code for execute the different prompts to the
OpenAI API.

3. `/retorch-llm-rp/src/main/resources`: contains all the necessary inputs for the prompts: scenarios, test cases and
3. `/llm-rp-expstudy/src/main/resources`: contains all the necessary inputs for the prompts: scenarios, test cases and
user requirements as well as the examples.

The naming conventions are :
Expand Down Expand Up @@ -48,8 +48,8 @@ the [retorch-st-fullteaching](https://github.com/giis-uniovi/retorch-st-fullteac

The user requirements are extracted of the FullTeaching documentation (Fuente Pérez, P. (2017). FullTeaching :
Aplicación Web de docencia con videoconferencia.) and translated to english. The spanish version can be
consulted [here](./retorch-llm-rp/src/main/resources/input/inputUserRequirements_spa.txt) and the english
version [here](./retorch-llm-rp/src/main/resources/input/inputUserRequirements_en.txt).
consulted [here](./llm-rp-expstudy/src/main/resources/input/inputUserRequirements_spa.txt) and the english
version [here](./llm-rp-expstudy/src/main/resources/input/inputUserRequirements_en.txt).

## Treatment Replication Overview

Expand All @@ -61,17 +61,17 @@ These two parts are detailed below.

- **Test Scenarios Generation:** This process is accomplished through the execution of a single script that take the
user requirements as input,
The output is provided in the resources (`retorch-llm-rp/src/main/resources/outputs`), namely with the version of the model and the
The output is provided in the resources (`llm-rp-expstudy/src/main/resources/outputs`), namely with the version of the model and the
prompting strategy used.

- **System Test Cases Generation:** The process takes the best previously generated test scenarios and several system
test cases as input. Automatically
the script makes a cross validation leaving the most close test case in terms of levenshtein distance, and asking the
model to generate its scenario.
The output is provided in the resources (`retorch-llm-rp/src/main/resources/outputs`), namely with the version of the model and the
The output is provided in the resources (`llm-rp-expstudy/src/main/resources/outputs`), namely with the version of the model and the
prompting strategy used and the scenario requested.

In both cases the prompts used are stored in the target folder (`retorch-llm-rp/src/main/resources/outputs`) for debugging purposes
In both cases the prompts used are stored in the target folder (`llm-rp-expstudy/src/main/resources/outputs`) for debugging purposes
The comparison baseline and how we selected the test cases from the original test suite is described in
the [Test Scenarios Baseline](./docs/RQ1-TestScenarios-Baseline.md)
and [Experimental Set-up](./docs/RQ2-ExperimentalSetup.md)
Expand Down
File renamed without changes.
2 changes: 1 addition & 1 deletion retorch-llm-rp/.project → llm-rp-expstudy/.project
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
<?xml version="1.0" encoding="UTF-8"?>
<projectDescription>
<name>retorch-llm-rp</name>
<name>llm-rp-expstudy</name>
<comment></comment>
<projects>
</projects>
Expand Down
4 changes: 2 additions & 2 deletions retorch-llm-rp/pom.xml → llm-rp-expstudy/pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -6,8 +6,8 @@
<artifactId>retorch-llm-rp</artifactId>
<version>1.0.0</version>
</parent>
<artifactId>retorch-llm-expstudy</artifactId>
<name>retorch-llm-rp</name>
<artifactId>llm-rp-expstudy</artifactId>
<name>llm-rp-expstudy</name>
<description>Replication package for 'Exploratory study of the usefulness of LLMs in System testing'</description>
<url>https://github.com/giis-uniovi/retorch-llm-rp</url>
<organization>
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,7 @@ public class ExperimentationHelper {

static final String PATH_KEY = "CHATGPT_API_KEY";
private final Logger log = LoggerFactory.getLogger(this.getClass());
private final String RESOURCES_ROUTE = "llm-rp-expstudy/src/main/resources";

public ExperimentationHelper() {
//This is the default constructor to avoid smells
Expand Down Expand Up @@ -93,7 +94,7 @@ public void sendChatGPTRequest(String body, String model, String experimentname)
// Format the current date and time
String formattedDateTime = now.format(formatter);

putOutputToFile("retorch-llm-rp/src/main/resources/outputs", experimentname + "-" + model + "-" + formattedDateTime, responseBody);
putOutputToFile(RESOURCES_ROUTE + "/outputs", experimentname + "-" + model + "-" + formattedDateTime, responseBody);
conn.disconnect();
}

Expand Down Expand Up @@ -149,7 +150,7 @@ public String getErrorResponse(InputStream i) throws IOException {
}

public String getTestCasesCrossValidation(int pos) throws IOException {
String[] rawTestCases = openFileLoadContent("retorch-llm-rp/src/main/resources/input/inputSystemTestCases.txt").split("//TC");
String[] rawTestCases = openFileLoadContent(RESOURCES_ROUTE+"/input/inputSystemTestCases.txt").split("//TC");
List<String> contentList = new ArrayList<>(Arrays.asList(rawTestCases));
contentList.remove(pos);

Expand All @@ -164,15 +165,15 @@ public String openFileLoadContent(String route) throws IOException {
}

public String getTestScenarios() throws IOException {
return openFileLoadContent("retorch-llm-rp/src/main/resources/input/inputTestScenarios.txt");
return openFileLoadContent(RESOURCES_ROUTE + "/input/inputTestScenarios.txt");
}

public String getUserRequirements() throws IOException {
return openFileLoadContent("retorch-llm-rp/src/main/resources/input/inputUserRequirements_en.txt");
return openFileLoadContent(RESOURCES_ROUTE + "/input/inputUserRequirements_en.txt");
}

public String getTestScenarioExample() throws IOException {
return openFileLoadContent("retorch-llm-rp/src/main/resources/input/inputTestScenarioExample.txt");
return openFileLoadContent(RESOURCES_ROUTE + "/input/inputTestScenarioExample.txt");
}

}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@

public class RQ1Experimentation {
private static final Logger log = LoggerFactory.getLogger(RQ1Experimentation.class);
private static final String OUT_BASE_PATH = "retorch-llm-rp/target/prompts-input/RQ1";
private static final String OUT_BASE_PATH = "llm-rp-expstudy/target/prompts-input/RQ1";
static ExperimentationHelper exHelper;

public static void main(String[] args) throws IOException {
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@

public class RQ2Experimentation {
private static final Logger log = LoggerFactory.getLogger(RQ2Experimentation.class);
private static final String OUT_BASE_PATH = "retorch-llm-rp/target/prompts-input/RQ2";
private static final String OUT_BASE_PATH = "llm-rp-expstudy/target/prompts-input/RQ2";
static ExperimentationHelper exHelper;

public static void main(String[] args) throws IOException {
Expand Down
2 changes: 1 addition & 1 deletion pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@
</properties>

<modules>
<module>retorch-llm-rp </module>
<module>llm-rp-expstudy</module>
</modules>

<dependencyManagement>
Expand Down

0 comments on commit a27e2a5

Please sign in to comment.