The Artemis Java Test Sandbox
Ares, also known as Artemis Java Test Sandbox (abbr. AJTS), is a JUnit 5 extension for easy and secure Java testing on the interactive learning platform Artemis.
Its main features are:
-
a security manager to prevent students crashing the tests or cheating
-
more robust tests and builds due to limits on time, threads and io
-
support for public and hidden Artemis tests, where hidden ones obey a custom deadline
-
utilities for improved feedback in Artemis like processing multiline error messages or pointing to a possible location that caused an Exception
-
utilities to test exercises using System.out and System.in comfortably
Project Status
Note: Ares requires at least Java 11.
Ares is provided as Maven/Gradle dependency. To use Ares in the test environment of a Maven project, include
<dependency>
<groupId>de.tum.in.ase</groupId>
<artifactId>artemis-java-test-sandbox</artifactId>
<version>1.13.0</version>
<scope>test</scope>
</dependency>in the dependencies section.
For Gradle projects, include
testImplementation 'de.tum.in.ase:artemis-java-test-sandbox:1.13.0'in the dependencies section.
You can now remove dependencies to JUnit 5, AssertJ and Hamcrest if present because Ares already includes them. If you want to use jqwik (>= 1.2.4) or JUnit 4 (JUnit 5 vintage), simply include them in the dependencies section.
Ares provides a high level of security, which comes at the cost of usability. Several steps need to be taken in order to make tests work properly, and it might require some time to understand what Ares does. Please study at least this complete basic usage guide before using Ares in production.
Assume you have a Java 11 Maven project, and the inside of pom.xml
looks like this:
<properties>
<maven.compiler.source>11</maven.compiler.source>
<maven.compiler.target>11</maven.compiler.target>
</properties>
<dependencies>
<dependency>
<groupId>de.tum.in.ase</groupId>
<artifactId>artemis-java-test-sandbox</artifactId>
<version>1.13.0</version>
<scope>test</scope>
</dependency>
</dependencies>Consider the following student class that needs to be tested:
import java.util.Objects;
public final class Penguin {
private final String name;
public Penguin(String name) {
this.name = Objects.requireNonNull(name, "name must not be null");
}
public String getName() {
return name;
}
}And you have already written the following simple JUnit 5 test class:
import static org.junit.jupiter.api.Assertions.*;
import org.junit.jupiter.api.Test;
public class PenguinTest {
@Test
void testPenguinPublic() {
Penguin pingu = new Penguin("Julian");
assertEquals("Julian", pingu.getName(), "getName() does not return the name supplied to the contructor");
}
@Test
void testPenguinHidden() {
assertThrows(NullPointerException.class, () -> new Penguin(null));
}
}In this example,
-
testPenguinPublic()is supposed to be executed after each push and directly give the students their feedback, while -
testPenguinHidden()should be executed only after the exercise deadline, and the results should not be visible before the deadline.
While Artemis has a feature to mark test cases as hidden, this will not prevent the contents of the test case leaking through static variables, files and similar, be it accidentally or on purpose. To prevent that, the hidden test case must not be executed before the deadline at all.
The public test case does not need to be hidden, as its purpose is to
give direct feedback. However, there are still multiple possible
problems like crashing the Maven build by System.exit(0) or containing
an endless loop. Both can have a negative impact on the interactive
learning experience because the students get confronted with an
incomprehensible log of a failed build. Such errors can be explained,
but that takes a lot of time, especially if it happens a lot (and it
will, if the number of students is sufficiently large).
It is also a security concern again, students could try to read the
.java files containing the test classes.
Therefore, we will use Ares to secure the tests and avoid unintelligible
feedback. The most basic way to do this is by using the @Public and
@Hidden annotations:
import static org.junit.jupiter.api.Assertions.*;
import org.junit.jupiter.api.Test;
// IMPORTANT: make sure to use the "jupiter" ones (if you are not using jqwik)
import de.tum.cit.ase.ares.api.jupiter.Hidden;
import de.tum.cit.ase.ares.api.jupiter.Public;
// This example won't work just like that, see below why
public class PenguinTest {
@Public
@Test
void testPenguinPublic() {
Penguin pingu = new Penguin("Julian");
assertEquals("Julian", pingu.getName(), "getName() does not return the name supplied to the contructor");
}
@Hidden
@Test
void testPenguinHidden() {
assertThrows(NullPointerException.class, () -> new Penguin(null));
}
}The code above won’t work just like that, if you try to run it as is,
you will get the following reported by JUnit:
java.lang.annotation.AnnotationFormatError: cannot find a deadline for hidden test testPenguinHidden()
Ares needs to know what the deadline is. We tell Ares with another annotation:
// Format must be ISO_LOCAL_DATE(T| )ISO_LOCAL_TIME( ZONE_ID)?
@Deadline("2020-06-09 03:14 Europe/Berlin")
public class PenguinTest {
// ...
}That annotation (like most of the Ares annotations) can also be placed on the test method (and nested classes), if multiple are present, the one that is closest to the test case is used.
Now, it already works! Try to play around with the deadline in the
annotation. If the given LocalDateTime lies in the past, the test case
is executed and - together with the student code presented earlier -
passes. If the deadline hasn’t passed, the test case won’t pass either.
It fails with
org.opentest4j.AssertionFailedError: hidden tests will be executed after the deadline.
and the test was not executed, as the deadline is always checked before
any hidden test case is executed.
You might have noticed that we specify the time zone as well. Although the annotation parser permits leaving it unspecified, this bears the risk of (not) executing the tests at the correct time if the build agent’s time zone is different from the one on your machine or what you would expect it to be. If you run tests where the time zone is/was not set, Ares will warn you about that in the logs.
The hidden test case was not executed and static variables cannot leak
its contents. If you change getName() to
public String getName() {
System.exit(0);
return name;
}You will now with Ares get the following error message:
java.lang.SecurityException: do not use System.exit(int) /// potential problem location: Penguin.getName(Penguin.java:12) ///
As you might be able to see, Ares threw a SecurityException. But it also
added
/// potential problem location: Penguin.getName(Penguin.java:12) ///.
This is the line from the stack trace which Ares thinks is most relevant
for the student, essentially, it searches for the uppermost stack frame
that is located in the student’s code. Student code is basically
everything that is not whitelisted.
But what is whitelisted?
-
The test class itself (in case of nested classes, the outermost class is whitelisted) and therefore, all its nested classes and methods, too.
-
A predefined set of packages, like everything that starts with
java.,sun.,com.sun.,org.junit,org.apache., … Therefore, never use such package names for student assignments! -
Single classes whitelisted using
@WhitelistClassand all classes matching@AddTrustedPackage -
Additional package prefixes provided with by system property
ares.security.trustedpackagesseparated by a comma,.
Ares also grants permissions that are requested by certain actions
(System.exit, File IO, Networking, Threads, …) based on whitelisted
stack frames. Ares granting a permission requires all stack frames to
be whitelisted.
Another test:
Adding one of the following lines to testPenguinPublic()
itself, and it will still pass using the correct student code:
Files.readString(Path.of("pom.xml"));
// or
Files.readString(Path.of("src/test/java/PenguinTest.java")); // assuming default maven structureIf you instead add one of the lines to the getName() method again, you
will get something like:
java.lang.SecurityException: access to path src\test\java\PenguinTest.java denied in line 16 in Penguin.java.
Which is exactly what you want, students should not be able to read the
code of the test classes. By default, student code has no access to any
path, not even read access.
By the way, adding @WhitelistClass(Penguin.class) to the test class or
method will make the test run fine again because Penguin is now
whitelisted and can therefore access all files without problems. So
never whitelist classes that students can edit.
Sadly, due to the way classes are loaded and the class path works when
testing student code with maven, there are still vulnerabilities if
students manage to load classes that would be in trusted packages. This
is especially problematic if they shadow library classes, such as
JUnit’s Assertions.
To prevent that, you have to use the Maven Enforcer Plugin to make sure no student content lands in trusted packages:
Maven:
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-enforcer-plugin</artifactId>
<version>3.1.0</version>
<executions>
<execution>
<id>enforce-no-student-code-in-trusted-packages</id>
<phase>process-classes</phase> <!--(1)-->
<goals>
<goal>enforce</goal>
</goals>
</execution>
</executions>
<configuration>
<rules>
<requireFilesDontExist>
<files>
<!--(2)-->
<file>${project.build.outputDirectory}/ch/qos/logback/</file>
<file>${project.build.outputDirectory}/com/github/javaparser/</file>
<file>${project.build.outputDirectory}/com/intellij/</file>
<file>${project.build.outputDirectory}/com/sun/</file>
<file>${project.build.outputDirectory}/de/tum/cit/ase/ares/api/</file>
<file>${project.build.outputDirectory}/java/</file>
<file>${project.build.outputDirectory}/javax/</file>
<file>${project.build.outputDirectory}/jdk/</file>
<file>${project.build.outputDirectory}/net/jqwik/</file>
<file>${project.build.outputDirectory}/org/apache/</file>
<file>${project.build.outputDirectory}/org/assertj/</file>
<file>${project.build.outputDirectory}/org/eclipse/</file>
<file>${project.build.outputDirectory}/org/jacoco/</file>
<file>${project.build.outputDirectory}/org/json/</file>
<file>${project.build.outputDirectory}/org/junit/</file>
<file>${project.build.outputDirectory}/org/opentest4j/</file>
<file>${project.build.outputDirectory}/sun/</file>
<file>${project.build.outputDirectory}/org/gradle/</file>
<file>${project.build.outputDirectory}/worker/org/gradle/</file>
</files>
</requireFilesDontExist>
</rules>
</configuration>
</plugin>Gradle:
def forbiddenPackageFolders = [ //(2)
"$studentOutputDir/ch/qos/logback/",
"$studentOutputDir/com/github/javaparser/",
"$studentOutputDir/com/intellij/",
"$studentOutputDir/com/sun/",
"$studentOutputDir/de/tum/cit/ase/ares/api/",
"$studentOutputDir/java/",
"$studentOutputDir/javax/",
"$studentOutputDir/jdk/",
"$studentOutputDir/net/jqwik/",
"$studentOutputDir/org/assertj/",
"$studentOutputDir/org/apache/",
"$studentOutputDir/org/eclipse/",
"$studentOutputDir/org/gradle/",
"$studentOutputDir/org/jacoco/",
"$studentOutputDir/org/json/",
"$studentOutputDir/org/junit/",
"$studentOutputDir/org/opentest4j/",
"$studentOutputDir/sun/",
"$studentOutputDir/worker/org/gradle/"
]
test {
doFirst { //(1)
for (String packageFolder in forbiddenPackageFolders) {
assert !file(packageFolder).exists(): "$packageFolder must not exist within the submission."
}
}
// ...
}-
Important: you want to enforce the non-existence of classes after their generation but before testing.
-
This is where all folders/packages go that we don’t want to exist in student code. You will always find the most recent recommendation for Ares here. If you use additional third-party libraries that need to be configured using
@AddTrustedPackage, you should add those packages here as well. Ares will check that all entries are present.
If you don’t want Ares to do so, set theares.maven.ignoreorares.gradle.ignoresystem property totrue. In case you want Ares to look into a different file, you can set theares.maven.pomorares.gradle.buildto a path other than the defaultpom.xmlorbuild.gradle.
Are we done now? With the most fundamental parts yes, but there is a bit more you need to know about testing with Ares, as this was just a very basic example with a single class and not much testing. Without further knowledge, you might not get Ares to work and consequently get rather annoyed or even enraged. To prevent that, please read on.
You can use @WhitelistPath and @BlacklistPath to control access to
paths. By default, no access is granted, and so you need to use
@WhitelistPath to give student code the permission to read and write
files explicitly. You can specify exceptions using @BlacklistPath
which will overpower the whitelisted paths.
The following examples will make use of course1920xyz as placeholder
value for the real Artemis exercise name/id. Replace it with the real
one when borrowing code snippets, or nothing will work as expected.
Most importantly, this does not only apply to explicit file IO, but also
to the .class files that the class loader reads, as needed. This
already happens if one student class requires another one, that has not
been loaded after that. You can recognize that in the standard error
output:
[WARN] [main] BAD PATH ACCESS: K:\repo\course1920xyz-solution\bin\some\Thing.class (BL:false, WL:false)
This usually means the class loader could not load the class. The parentheses show, that the problem is the missing whitelisting. Therefore, all test setups should have some whitelisting.
A number of examples how you can whitelist paths in Ares:
-
@WhitelistPath("")will grant read access to the paths in the directory of execution, which is usually where thepom.xmlis. -
@WhitelistPath("pom.xml")will allow students to read thepom.xml. -
@WhitelistPath("..")will allow read access to the level above the maven project. In Eclipse, that is the level of your workspace. -
@WhitelistPath(value = "../course1920xyz**", type = PathType.GLOB)grants read access to projects beginning with the exercise "id" used by Artemis. Should you use the Eclipse feature "Referenced Projects" (or the analog to that in your IDE) to link the student/solution project to the tests, you will need a setting like this. -
@WhitelistPath(value = "data", level = PathActionLevel.DELETE)will allow students to read, write and delete files in thedatadirectory and subdirectories. -
@WhitelistPath("target")allows reading files in target (Maven output folder) -
@BlacklistPath(value = "*Test.{java,class}", type = PathType.GLOB)prevents access to classes in source code or compiled form that containTest. If you leave away the*afterTest, nested classes are not blacklisted. Student classes should not be called something with "Test" then.
That was not everything but already quite a lot. Take a look at the Javadoc of the annotations and enums used, if you want to know more. Before you give up, here is my recommendation how to start:
@WhitelistPath(value = "../course1920xyz**", type = PathType.GLOB) // for manual assessment and development
@WhitelistPath("target") // mainly for Artemis
@BlacklistPath("target/test-classes") // prevent access to test-related classes and resourcesAdd a @BlacklistPath for other important classes, like your reference
implementations of the solution to test against should you use that
approach.
Note: the Artemis project starts with course1920xyz, but the build in
Bamboo (by Artemis) will happen in a directory named after the build
plan, which is in upper case and therefore, begins with COURSE1920XYZ.
Make sure that you do not build multiple student solutions in the same
directory on the same machine using the git clone (lower case) approach.
Otherwise, adjust the whitelisting to your needs.
JUnit already provides means of applying timeouts to tests. However, those are not strict in the sense of "enforced in the strongest possible way". What is meant by that?
There are three different ways how the timeouts can work:
-
like
org.junit.jupiter.api.Timeout
This timeout is not preemptive, and the test itself runs in the same thread executing the tests. It will only try to stop the test via an interrupt. If that fails like it does for an endless loop, the test will definitively fail. After it is finished. Which might never happen and the main reason not to use this when it comes to testing unknown code. -
like
org.junit.jupiter.api.Assertions.assertTimeoutPreemptively
This will fail the test preemptively by executing theExecutableargument itself in a different thread than the thread executing all tests. It will only try to stop the test via an interrupt, but if that fails it will simply carry on. The test thread might still run, though. -
like
de.tum.cit.ase.ares.api.StrictTimeout
This uses a mechanism similar toassertTimeoutPreemptively, but will resort to harder means if necessary. It will in the following order:-
wait the given duration
-
interrupt the thread executing the test and wait no longer (like
assertTimeoutPreemptively) -
block the creation of new threads
-
interrupt all threads created during the test and try to join the threads
-
if that fails, use
Thread.stop()on all remaining threads and try to join again -
repeat step 5 multiple times, if required
-
Should that fail, report a special SecurityException that not all threads could be stopped. (see the standard error output for a detailed report then) If that happens, no more tests can be properly executed because the security cannot be guaranteed and the test cases cannot be executed "in isolation". All following tests will fail.
-
Rule 1: When testing with Ares, always use @StrictTimeout for
timeouts, the others will not work reliably, especially in conjunction
with the Ares security.
Rule 2: When writing tests for Artemis, always use @StrictTimeout.
There is no reason to omit the timeout, since you do not know the code
students will write. (And they will write code spawning millions of
threads in endless loops, which in turn will do the same recursively.)
By default, Ares will record standard and error output of each test
internally and not print it to the console. The recorded output can then
be obtained and tested, see
IOTester The reason for this is on
the one hand to keep the console and logs short and clean and on the
other hand prevent students from accidentally messing up the logs with
millions of lines. Ares also has a hard limit on the total number of
printed chars at around 10 million.
To mirror the output recorded by Ares to the console, use the
@MirrorOutput annotation on the test class or method.
It is also worth noting that Ares enforces valid UTF-8 being printed and throws an appropriate exception otherwise.
Hidden tests will be executed by Ares only after the deadline. This poses the problem, how the exercise creators should work on the tasks, tests and the sample solution. One possible solution would be to use an alternative deadline annotation or change the deadline temporarily. The problem is that it is quite likely one might forget to change it back again, and protecting the hidden tests would fail.
Use @ActivateHiddenBefore just like @Deadline to state the
LocalDateTime before which hidden tests should be executed. This date
should, of course, be before the release of the exercise on Artemis.
You can use @ExtendedDeadline together with a duration like 1d or
2d 12h 30m to extend the deadline by the given amount.
@ExtendedDeadline("1d"), for example, extends the deadline by one day.
If you use the annotation on different levels (e.g. class and method)
without stating a new deadline (e.g. deadline only on class level), the
extensions will be added together.
By default, Ares will not allow non-whitelisted code to use threads at
all. That includes thread pools, but excludes the common pool and its
users, like parallel streams. To allow the use of Threads, use the
annotation @AllowThreads. The number of active threads is also
limited, the default value of that is 1000, but can be changed in the
annotation. Please keep in mind that this limit should not be larger
than 1000 to prevent performance and timeout chaos.
New threads are for security reasons not directly whitelisted by Ares
and will not be allowed to do anything security critical. If you trust a
thread (at least its entry point), you can explicitly request the thread
to be whitelisted using
ArtemisSecurityManager.requestThreadWhitelisting(Thread). The thread
calling the method and its stack must be whitelisted, of course.
One example showing some possibilities here:
void testSquareCorrect(IOTester tester) { //(1)
tester.provideInputLines("5"); //(2)
InputOutputPenguin.calculateSquare(); //(3)
tester.err().assertThat().isEmpty(); //(4)
tester.out().assertThat().isEqualTo("""
Enter Number:
Answer:
25"""); //(5)
}-
Declare
IOTesteras parameter. -
Provide input lines before calling the student code. This content will be used for reading lines from
System.in. -
Call the student code to process the input and produce output.
-
Assert that nothing was printed to
System.err. -
Assert that the standard output (in this case excluding the final line break) is equal to the given text block (if you use text blocks, be aware of their newline handling).
Note that Ares normalizes the line breaks to \n, and
OutputTester
offers many different approaches to checking output (e.g. single string, list of strings, …).
If students read more lines than provided, they get the following feedback: … java.lang.IllegalStateException: no further console input request after the last(number 1: "5") expected. …
See also IOTester and for more examples, the
InputOutputUser
test.
Showing Standard Output covers how the student output is managed and shown in the test logs.
|
Tip
|
In case the default |
Ares allows for local network connections by using the @AllowLocalPort
annotation.
There are plenty of configuration options, and the code can get complicated quickly due to the threads required to test network connections. One issue can be that waiting network connections block threads in such a way that they cannot be stopped (waiting in native code), so we recommend using timeouts for connections at least on one end consistently.
For examples, have a look at the test
NetworkUser.
You can set a locale for Ares (and the rest of Java) by adding the
@UseLocale JUnit extension to classes/methods, which will set the Java
default locale
that is also used by Ares. The locale is changed only for the scope
where the annotation is applied.
Ares is currently localized in German (de_DE) and English (en_US),
where en_US is the fallback for any other locale.
See also the LocaleUser
test for more examples.
For versions prior to 1.0.0, a repository block had to be added to
<repositories> section of the pom.xml that referenced the Maven
repository URL https://gitlab.com/ajts-mvn/repo/raw/master/.
Using older Ares versions is highly discouraged, remove these repository declarations and update to the newest Ares version if they appear in your projects.
GitHub Packages does currently not allow unregistered, public access to the packages. Therefore, you will need to authenticate to GitHub if you use
<repositories>
<repository>
<id>ares</id>
<name>Ares Maven Packages</name>
<url>https://maven.pkg.github.com/ls1intum/Ares</url>
</repository>
</repositories>Ares was created by Christian Femers and is licensed under the
MIT License, see
LICENSE.md.