Skip to content

Latest commit

 

History

History
136 lines (96 loc) · 12.8 KB

TESTING.md

File metadata and controls

136 lines (96 loc) · 12.8 KB
Raw

Testing

Code Coverage Collecting

Code coverage is a measurement of how many lines of code are executed under the test suites.

OpenSearch plugin

It is recommended to use JaCoCo library to generate the code coverage report. The following guide may help you use JaCoCo in OpenSearch plugins with ease.

Gradle project
  1. Add JaCoCo Gradle plugin to build.gradle in your repository. example
  2. Set Jacoco to export coverage report in xml format. example
  3. Call gradle task "jacocoTestReport" to generate report (./gradlew jacocoTestReport) after running your tests. (Or add a dependsOn to make "jacocoTestReport" task depending on your test tasks. example)
Maven project

Since OpenSearch uses Gradle as the build tool, we don't recommend using Maven. Code coverage for a Maven project can be generated by JaCoCo Maven plugin, and here is an example.

Limitation
  1. JaCoCo Gradle plugin can not collect code coverage properly for integration tests (tests that extend OpenSearchIntegTestCase or OpenSearchRestTestCase, see here to learn more). However, there is a workaround to collect coverage when the integration test is running on a local single node cluster. For your reference, this file is a custom Gradle plugin to help achieve, and it can be applied conditionally like here.
  2. Additional work is needed to get coverage report by JaCoCo for the test classes which use PowerMock, because those classes must be instrumented before the test executes. Please see the PowerMock wiki for detail, and there is an example for using JaCoCo Maven plugin. But for JaCoCo Gradle plugin, there is no native support, see the issue in gradle repository for detail.
  3. JaCoCo Gradle plugin can not merge code coverage reports from different sub-projects out-of-box, when the code repository contains a Gradle multi-project build. If you need the aggrectiated code coverage report from multiple sub-projects, please check the example of a custom Gradle plugin in OpenSearch Alerting plugin, or the sample from Gradle.

OpenSearch Dashboards plugin

OpenSearch Dashboards mainly uses Jest for unit testing, and plugins are recommended to utilize Jest as the unit testing framework.

To generate code coverage report under Jest framework, adding the flag --coverage in the command to run unit tests. See here for an example in the CI workflow.

Code Coverage Reporting

OpenSearch/Dashboards plugins are using Codecov for code coverage reporting and analysis. The dashboard can be seen here.

All the OpenSearch plugins are required to have the following items:

Code coverage report upload through CI workflow

  1. Generate code coverage report in xml format in a CI workflow of Github Actions
  2. Upload the report to Codecov by adding a step of codecov-action in the workflow.
  3. For private repositories, an upload token is needed, visit the repository page of Codecov to get your token: https://codecov.io/gh/opensearch-project/REPO-NAME-HERE. Please follow Github docs to use Github Secrets to store the token and not expose to the public.
  4. Set the workflow to be triggered by push and pull request on main and release branches.
  5. To see your code coverage results from Codecov, visiting https://codecov.io/gh/opensearch-project/REPO-NAME-HERE after the CI workflow is triggered.

See CI Workflows for example.

A status badge in README file to show the code coverage

See here for an example of the status badge.

Add the following line at the header of the README markdown file of the repository:

[![codecov](https://codecov.io/gh/opensearch-project/REPO-NAME-HERE/branch/main/graph/badge.svg)](https://codecov.io/gh/opensearch-project/REPO-NAME-HERE)

Link of the badge can also be accessed from the repository's settings page of Codecov: https://app.codecov.io/gh/opensearch-project/REPO-NAME-HERE/settings/badge

Codecov integration in Pull Requests

  1. Add a codecov.yml file into the repository (example)
  2. Set the target code coverage properly in case it blocks your PR in Github checks.

See Codecov docs to learn more about codecov yaml file and common configurations.

Backwards Compatibility Testing

The purpose of backwards compatibiity tests in plugins is to ensure that the plugin is safely upgraded when an OpenSearch node undergoes an upgrade. Backwards compatibility tests can be implemented in plugins by extending the OpenSearch bwc framework.

See OpenSearch#1051 for more information on the extended bwc framework.

Types of BWC tests

  1. Mixed cluster test: This tests a cluster running with three nodes of a bwc OpenSearch and plugin version and upgrades a single node to the current version of OpenSearch and the plugin while the other two nodes are still on the previous version. This creates a mixed cluster. The test checks that the correct version of the plugin is installed in the upgraded node and the plugin functionalities work as expected after the upgrade. Command: ./gradlew adBwcCluster#mixedClusterTask -Dtests.security.manager=false
  2. Rolling upgrade tests: This tests a cluster running with three nodes of a bwc OpenSearch and plugin version and performs a node by node upgrade to the current version of OpenSearch and the plugin. The test checks that the correct version of the plugin is installed in each upgraded node and the plugin functionalities work as expected after each node upgrade. Commmand: ./gradlew adBwcCluster#rollingUpgradeClusterTask -Dtests.security.manager=false
  3. Full restart upgrade tests: This tests a cluster running with three nodes of a bwc OpenSearch and plugin version and performs a full cluster upgrade for all the nodes to the current version of OpenSearch and the plugin. The test checks that the correct version of the plugin is installed in all upgraded nodes and the plugin functionalities work as expected after the upgrade. Command: ./gradlew adBwcCluster#fullRestartClusterTask -Dtests.security.manager=false
  4. Command: ./gradlew bwcTestSuite -Dtests.security.manager=false is used to run all the above bwc tests together.

The logs for the test runs can be found at build/testclusters/{clustername}-{nodeid}/logs.

As mentioned above, the bwc tests are using the extended bwc framework from OpenSearch - the test cluster is contained within the testclusters module which is available as a plugin opensearch.testclusters. Each cluster is an instance of OpenSearchCluster containing a given number of OpenSearchNode nodes. The OpenSearchNode allows to install plugins to the node, start, stop, upgrade (along with plugin upgrade) and restart the node. The plugin bwc tests use these functionalities to spin up a test cluster with provided bwc versions and upgrade the nodes and plugins installed on the nodes to the current version using upgradeNodeAndPluginToNextVersion for a single node at a time and upgradeAllNodesAndPluginsToNextVersion for a full restart upgrade.

See anomaly-detection#158 and anomaly-detection#185 for more information.

Each plugin can test various functionalities in their bwc tests and add more scenarios if needed.

BWC tests on distribution bundle level

The BWC tests running on distribution level are using the same framework from OpenSearch. The test cluster is spin up with the latest distribution bundle of provided version exclusively when the project is initialized with property -PcustomDistributionDownloadType=bundle.

The distribution bundle will be obtained with this format URL: https://ci.opensearch.org/ci/dbc/distribution-build-opensearch/1.3.2/latest/linux/x64/tar/dist/opensearch/opensearch-1.3.2-linux-x64.tar.gz This URL pattern starts to be supported as far as v1.3.2 so The distribution bundle set up in the test cluster can be all versions after version v1.3.2.

On CI level for plugins, security certificates need to be manually imported when spinning up the test cluster as security plugin is included in the distribution bundle. When upgrading the version within the test cluster, nextNodeToNextVersion is used for a single node upgrade and goToNextVersion is for a full restart upgrade.

See anomaly-detection#766 or observability#1366 for more information.

Hooking the BWC tests to CI

The bwc tests can be hooked to the CI workflow in the plugin to run it with each pull request and push.

See anomaly-detection#181 for more information.

BWC Version test matrix

The bwc tests should follow the OpenSearch's support model, in this way the last major version's most recent minor version should be tested. Additionally all minor versions of the current major version should also be tested.

At the time of this writing there are OpenSearch releases for 1.0, 1.1, 1.2, 1.3, and 2.0. To ensure bwc tests have appropriate coverage when building components for 2.1, the bwc test matrix should include 1.3, and 2.0.

When minor build version number bumps are made the bwc test matrix should include all previous minor versions of the same major version.

When major build version number bumps are made the bwc test matrix will remove the previous major version, and only support the single most recently released major & minor version.

Gradle Plugins

Distribution Download Plugin

The Distribution Download plugin downloads the latest version of OpenSearch by default, and supports overriding this behavior by setting customDistributionUrl. This will help to pull artifacts from custom location for testing during release process.

./gradlew integTest -PcustomDistributionUrl="https://ci.opensearch.org/ci/dbc/bundle-build/1.2.0/1127/linux/x64/dist/opensearch-1.2.0-linux-x64.tar.gz"