Skip to content

Reproducing Test Failures Locally

Jump to bottom
Lan Xia edited this page Apr 26, 2019 · 36 revisions

General steps

  1. Ensure test machine is set up with test prereqs.

  2. Build or download/unpack the SDK to /someLocation

  3. git clone https://github.com/AdoptOpenJDK/openjdk-tests.git (to /testLocation)

  4. cd openjdk-tests

  5. ./get.sh -t /testLocation/openjdk-tests -p < somePlatform > -i < someImpl >

  6. cd TestConfig

  7. export environment variables suitable for the SDK under test

  8. make -f run_configure.mk // generates test targets applicable to the platform, implementation and version

  9. make compile // fetches test material and compiles it, based on build.xml files in the test directories

  10. make _< someTestTarget > // executes the test target (can be test group, level, level.group or specific test). i.e., openjdk (all tests in openjdk group), sanity.functional (all functional tests labelled at sanity level), extended.system (all system tests labelled at extended level), jdk_math (the specific jdk_math target defined as part of openjdk group), MauveMultiThreadLoadTest_0 (the first variation of the specific system test called MauveMultiThreadLoadTest), etc

Degrees of Freedom in Testing

Below is a list of common scenarios. As you can see, the approach to running the different groups of tests is very similar. With minimal changes to environment variables and target names, a great variety of test scenarios can be run. These instructions can be used for running any group of tests on any JDK version, any JDK implementation, and any platform/spec. For more details, please see the OpenJ9 Test Documentation.

There are a number of degrees of freedom by which you can generate, compile and execute test targets. Some variants are:

  • test group [functional|system|openjdk|external|perf]
  • level [sanity|extended|special]
  • jdk version - [8|9|10|11]
  • jdk impl - [openj9|hotspot|sap|ibm]
  • platform - [x64_linux|x64_mac|s390x_linux|ppc64le_linux|aarch64_linux|ppc64_aix]
  • spec - [linux_x86-64_cmprssptrs|linux_x86-64| etc] - as listed in buildspecs

JDK_VERSION, JDK_IMPL and SPEC can be auto-detected by our test framework. They are no longer required.

If needed, users can turn off the auto-detection by setting AUTO_DETECT=off. If users set AUTO_DETECT off, then JDK_VERSION, JDK_IMPL and SPEC values are required.

Specific Test recipes (demonstrative, not an exhaustive list)

Test group JDK version JDK impl Platform/Spec get.sh options Env variables
functional 8 openj9 x64_linux/linux_x86-64_cmprssptrs -p x64_linux -i openj9 export TEST_JDK_HOME=/someLocation
export BUILD_LIST=functional
system 10 openj9 x64_linux/linux_x86-64_cmprssptrs -p x64_linux -i openj9 export TEST_JDK_HOME=/someLocation
export BUILD_LIST=systemtest
openjdk 11 hotspot x64_mac/macos_x86-64 -p x64_mac -i hotspot export TEST_JDK_HOME=/someLocation
export BUILD_LIST=openjdk_regression
openjdk 11 openj9 s390x_linux/linux_390-64_cmprssptrs -p s390x_linux -i openj9 export TEST_JDK_HOME=/someLocation
export BUILD_LIST=openjdk_regression
export JDK_CUSTOM_TARGET=java/lang/Class/GetModuleTest.java
external 8 openj9 x86_linux/linux_x86-64_cmprssptrs -p x64_linux -i openj9 export TEST_JDK_HOME=/someLocation
export BUILD_LIST=thirdparty_containers
perf 11 openj9 x86_linux/linux_x86-64_cmprssptrs -p x64_linux -i openj9 export TEST_JDK_HOME=/someLocation
export BUILD_LIST=performance

Functional tests

Run extended functional tests on jdk8_x86-64_linux_openj9 SDK

  • Build or unpack the jdk8_x86-64_linux_openj9 SDK to /someLocation
  • git clone https://github.com/AdoptOpenJDK/openjdk-tests.git (to /testLocation)
  • cd openjdk-tests
  • ./get.sh -t /testLocation/openjdk-tests -p x64_linux -i openj9
  • cd TestConfig
  • export environment variables matching the SDK under test ( export TEST_JDK_HOME=/someLocation export BUILD_LIST=functional )
  • make -f run_configure.mk
  • make compile
  • make _extended.functional

Run the sanity tests in the functional/JLM_Tests directory on jdk10_x86-64_linux_openj9 SDK

  • Build or unpack the jdk10_x86-64_linux_openj9 SDK to /someLocation
  • git clone https://github.com/AdoptOpenJDK/openjdk-tests.git (to /testLocation)
  • cd openjdk-tests
  • ./get.sh -t /testLocation/openjdk-tests -p x64_linux -i openj9
  • cd TestConfig
  • export environment variables matching the SDK under test ( export TEST_JDK_HOME=/someLocation export BUILD_LIST=functional )
  • make -f run_configure.mk
  • make compile
  • make -C ../functional/JIT_Test -f autoGen.mk _sanity

System tests

Run sanity system tests on jdk10_x86-64_linux_openj9 SDK

  • Build or unpack the jdk10_x86-64_linux_openj9 SDK to /someLocation
  • git clone https://github.com/AdoptOpenJDK/openjdk-tests.git (to /testLocation)
  • cd openjdk-tests
  • ./get.sh -t /testLocation/openjdk-tests -p x64_linux -i openj9
  • cd TestConfig
  • export environment variables matching the SDK under test ( export TEST_JDK_HOME=/someLocation export BUILD_LIST=systemtest )
  • make -f run_configure.mk
  • make compile
  • make _sanity.system

OpenJDK tests

Run sanity openjdk tests on jdk11_x64_macos_hotspot SDK

  • Build or unpack the jdk11_x64_macos_hotspot SDK to /someLocation
  • git clone https://github.com/AdoptOpenJDK/openjdk-tests.git (to /testLocation)
  • cd openjdk-tests
  • ./get.sh -t /testLocation/openjdk-tests -p x64_mac -i hotspot
  • cd TestConfig
  • export environment variables matching the SDK under test ( export TEST_JDK_HOME=/someLocation export BUILD_LIST=openjdk_regression )
  • make -f run_configure.mk
  • make compile
  • make _sanity.openjdk

Run specific test class from openjdk tests on jdk11-linux-s390x-openj9 SDK

For this example, let's run a currently excluded test, java/lang/Class/GetModuleTest.java found in exclude file

  • Build or unpack the jdk11-linux-s390x-openj9 SDK to /someLocation
  • git clone https://github.com/AdoptOpenJDK/openjdk-tests.git (to /testLocation)
  • cd openjdk-tests
  • ./get.sh -t /testLocation/openjdk-tests -p s390x_linux -i openj9
  • cd TestConfig
  • export environment variables matching the SDK under test ( export TEST_JDK_HOME=/someLocation export BUILD_LIST=openjdk_regression export JDK_CUSTOM_TARGET=java/lang/Class/GetModuleTest.java )
  • make -f run_configure.mk
  • make compile
  • make _jdk_custom

External tests

Run scala tests in external group on jdk8-x86-64_linux_openj9 SDK (in Docker container)

  • Build or unpack the jdk8-x86-64_linux_openj9 SDK to /someLocation
  • git clone https://github.com/AdoptOpenJDK/openjdk-tests.git (to /testLocation)
  • cd openjdk-tests
  • ./get.sh -t /testLocation/openjdk-tests -p x64_linux -i openj9
  • cd TestConfig
  • export environment variables matching the SDK under test ( export TEST_JDK_HOME=/someLocation export BUILD_LIST=thirdparty_containers )
  • make -f run_configure.mk
  • make compile
  • make _scala_test

Perf tests

Run idle_micro bench in perf group on jdk11-x86-64_linux_openj9 SDK

  • Build or unpack the jdk11-x86-64_linux_openj9 SDK to /someLocation
  • git clone https://github.com/AdoptOpenJDK/openjdk-tests.git (to /testLocation)
  • cd openjdk-tests
  • ./get.sh -t /testLocation/openjdk-tests -p x64_linux -i openj9
  • cd TestConfig
  • export environment variables matching the SDK under test ( export TEST_JDK_HOME=/someLocation export BUILD_LIST=performance )
  • make -f run_configure.mk
  • make compile
  • make _IdleMicrobenchmark_J9
Clone this wiki locally