-
Notifications
You must be signed in to change notification settings - Fork 77
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
docs(run groups): improved images and code samples (#3868)
* docs(rungroups): add bigger images * docs(run groups): add bigger image and code sample * docs(run groups): complete all ACs * docs(recipes): fix test files
- Loading branch information
Adnan Rahić
authored
May 24, 2024
1 parent
645d59f
commit 82664db
Showing
8 changed files
with
218 additions
and
171 deletions.
There are no files selected for viewing
This file was deleted.
Oops, something went wrong.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,178 @@ | ||
--- | ||
id: run-groups | ||
title: Run Groups | ||
description: Tracetest uses the concept of Run Groups that encapsulate one or more runs. Each time a test run is executed in the Tracetest app, a run group is generated that can be found in the UI. | ||
keywords: | ||
- tracetest | ||
- trace-based testing | ||
- observability | ||
- distributed tracing | ||
- testing | ||
- test runs | ||
- test run groups | ||
image: https://res.cloudinary.com/djwdcmwdz/image/upload/v1698686403/docs/Blog_Thumbnail_14_rsvkmo.jpg | ||
--- | ||
|
||
:::tip Commercial Feature | ||
[Feature available only in Cloud-based Managed Tracetest & Enterprise Self-hosted Tracetest.](https://tracetest.io/pricing) | ||
::: | ||
|
||
Run Groups are containers that encapsulate one or more runs. Each time a test run is executed in the Tracetest app, a run group is generated that can be found in the UI. | ||
|
||
![Run Groups UI](../img/run-groups-ui.png) | ||
|
||
Clicking on a run group will show the runs the run group contains: | ||
|
||
![Run Groups Detail](../img/run-groups-detail.png) | ||
|
||
Run Groups can be used to logically group runs that have context together. This allows specifying a series of tests or test suites to run concurrently. You then wait for the entire collection to pass or fail before making a decision if the entire group succeeded. | ||
|
||
:::note | ||
You can asynchronously run multiple tests from a script in one run group and wait for all of them to finish before deciding whether to deploy a new release. | ||
::: | ||
|
||
Run Groups track the status of all test suites and test runs to yield a single result. | ||
|
||
Users are allowed to attach more runs to a run group by adding them in the run information that’s available in the CLI. | ||
|
||
[Integrations](https://docs.tracetest.io/tools-and-integrations/overview) support this feature by creating a single run group for every execution. | ||
- [k6](https://docs.tracetest.io/tools-and-integrations/k6) | ||
- [Playwright](https://docs.tracetest.io/tools-and-integrations/playwright) | ||
- [Cypress](https://docs.tracetest.io/tools-and-integrations/cypress) | ||
- [Artillery](https://docs.tracetest.io/tools-and-integrations/artillery-plugin) | ||
|
||
Run groups are essential for integrations! Without it, a k6 test could potentially generate thousands of test runs that would pollute the environment and make it almost impossible to search for test runs not associated with that k6 test run. | ||
|
||
Another good use case for run groups is for CI runs: usually we don't run a single test or a single test suite to test our systems. You can expect some users to have dozens of suites being run during a CI pipeline. With run groups, all the suite runs from the same pipeline will be shown on the same screen, making it easier for users to understand what failed. | ||
|
||
Last, but not least important, if multiple tests are run inside a run group, they will run in parallel and finish faster than if you use a test suite to organize your tests. (Test suites run sequentially, so for the next test to be run, the current test must finish executing, which can be pretty time consuming.) | ||
|
||
## Using Run Groups from the CLI | ||
|
||
:::note | ||
[Check out the sample code on GitHub here.](https://github.com/kubeshop/tracetest/tree/main/examples/environment-automation) | ||
::: | ||
|
||
You can run multiple tests at once by providing multiple `-f file.yaml` parameters to the `tracetest run test` command. These tests will be placed under the same run group. | ||
|
||
You can also manually wait for a run group to be done from the CLI. | ||
|
||
View the [sample code](https://github.com/kubeshop/tracetest/tree/main/examples/environment-automation). You will see multiple test definitions. | ||
|
||
```yaml title="facebook.yaml" | ||
type: Test | ||
spec: | ||
id: facebook-test | ||
name: Facebook | ||
trigger: | ||
type: http | ||
httpRequest: | ||
method: GET | ||
url: facebook.com | ||
headers: | ||
- key: Content-Type | ||
value: application/json | ||
skipTraceCollection: true | ||
``` | ||
```yaml title="google.yaml" | ||
type: Test | ||
spec: | ||
id: google-test | ||
name: Google | ||
trigger: | ||
type: http | ||
httpRequest: | ||
method: GET | ||
url: google.com | ||
headers: | ||
- key: Content-Type | ||
value: application/json | ||
skipTraceCollection: true | ||
``` | ||
Run these two tests in a run group like this: | ||
```bash | ||
tracetest run test -f facebook.yaml -f google.yaml -f file.yaml -f ... | ||
``` | ||
|
||
Normally, if you run two tests separately, both test runs would be part of different run groups: | ||
|
||
```bash | ||
tracetest run test -f test1.yaml | ||
``` | ||
|
||
```bash | ||
tracetest run test -f test2.yaml | ||
``` | ||
|
||
You also can specify the run group id when running a test. This would make the test part of the run group. | ||
|
||
```bash | ||
tracetest run test -f test1.yaml --group my-pipeline | ||
tracetest run test -f test2.yaml --group my-pipeline | ||
``` | ||
|
||
With the `--group` argument, both tests would be run in parallel but would be seen inside the same run group. | ||
|
||
The CLI will by default wait for the group to finish running. Use the `-W` or `--skip-result-wait` flags to not wait for results and exit immediately after the run group has started. | ||
|
||
## Running multiple tests inside the same group | ||
|
||
```bash | ||
tracetest run test -f test1.yaml -f test2.yaml | ||
``` | ||
|
||
Both test runs will be shown on the same run group (with a random id). | ||
|
||
## Running multiple tests inside the same group and choosing the group id | ||
|
||
```bash | ||
tracetest run test -f test1.yaml -f test2.yaml --group my-group-id | ||
``` | ||
|
||
Both tests will be shown on the same run group (id = my-group-id). If this run group already exists, the two new runs will be placed there with any other test run from that group. | ||
|
||
## Adding a new run to an existing group | ||
|
||
```bash | ||
tracetest run test -f test3.yaml --group my-group-id | ||
``` | ||
|
||
This new run will be part of the `my-group-id` group along with the `test1` and `test2` runs. | ||
|
||
## Running multiple tests via `id` inside the same group and choosing the group id | ||
|
||
```bash | ||
tracetest run test --id testid1 --id testid2 --group my-group-id | ||
``` | ||
|
||
You can also use the `--id` of a test saved in Tracetest instead of a test definition file. | ||
|
||
## Running a test suite and multiple tests inside the same run group | ||
|
||
```bash | ||
tracetest run test -f suite.yaml -f test1.yaml -f test2.yaml | ||
``` | ||
|
||
Test suites can also be part of a run group along with other suites or tests. | ||
|
||
## Difference between a test suite and a run group | ||
|
||
![](https://res.cloudinary.com/djwdcmwdz/image/upload/v1716551731/docs/mermaid-js.github.io_mermaid-live-editor__view_lj7fe6.png) | ||
|
||
### Execution time | ||
|
||
A test suite only executes the next test when the current test has finished running. So if it contains 4 tests that take 1 minute each, the whole suite will take 4 minutes to complete. Run group tests are run in parallel, so if the same 4 tests are run inside a group, it will complete in 1 minute. | ||
|
||
### Using data from a previous test | ||
|
||
In suites, you can chain your tests and share data from one test to another. For example, getting the response from the first test and use it as part of the second test definition. This is not possible with run groups. | ||
|
||
### Composition | ||
|
||
Suites can only contain tests inside of it. Groups can contain both tests and suites. This means that if you have 5 tests where 3 are totally independent but the other 2 need each other to work, you can use groups to run those tests in an efficient way: | ||
|
||
- Running the suite (with 2 tests) and the 3 independent tests in parallel. | ||
- If all tests take 1 minute to run, the run group would take 2 minutes to run. |
This file was deleted.
Oops, something went wrong.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,24 @@ | ||
--- | ||
id: runs | ||
title: Runs | ||
description: Tracetest uses the concept of Runs to define every time a Test was triggered. Runs from different Tests can be grouped into Run Groups to be triggered in parallel. | ||
keywords: | ||
- tracetest | ||
- trace-based testing | ||
- observability | ||
- distributed tracing | ||
- testing | ||
- test runs | ||
- test run groups | ||
image: https://res.cloudinary.com/djwdcmwdz/image/upload/v1698686403/docs/Blog_Thumbnail_14_rsvkmo.jpg | ||
--- | ||
|
||
Tracetest uses the concept of Runs to define every time a Test was triggered. | ||
|
||
- [Tests](/concepts/tests) | ||
|
||
Runs from different Tests can be grouped into Run Groups to be triggered in parallel. | ||
|
||
- [Run Groups](/concepts/run-groups) | ||
|
||
![Runs](https://res.cloudinary.com/djwdcmwdz/image/upload/v1716550299/docs/app.tracetest.io_organizations_ttorg_2179a9cd8ba8dfa5_environments_ttenv_597cf82c754c2a01_runs_page_1_name_startDate_endDate_ytxhk1.png) |
Oops, something went wrong.