Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement titlePath to improve the default hierarchy of test results #1260

Open
delatrie opened this issue Feb 20, 2025 · 0 comments
Open

Implement titlePath to improve the default hierarchy of test results #1260

delatrie opened this issue Feb 20, 2025 · 0 comments
Labels
theme:commons type:new feature New feature or request

Comments

@delatrie
Copy link
Collaborator

delatrie commented Feb 20, 2025

Is your feature request related to a problem? Please describe.
Allure 2 comes with three built-in hierarchies: suites, bdd, and packages. Each comes with some drawbacks.

package is a well-known label that is usually automatically filled by Allure integrations. When opening the report, Allure splits the package values by . and groups them into a tree. Each package value therefore is a path to the node containing the test result in that tree.

The drawbacks:

  • It groups tests on a higher level (e.g., on the file or the package levels, depending on the language and the framework).
  • It's a label, which means the value is stored as a single string. The path is restored by Allure by splitting the value by .. If a path fragment (e.g., a filename) contains ., it might lead to unexpected results.

suite and bdd are based on two sets of labels: parentSuite, suite, and subSuite and epic, feature, and story. Each label corresponds to one level of the corresponding hierarchy. Many integrations fill one of the hierarchies, leaving the other to a user.

The drawbacks:

  • It's not always clear which hierarchy a specific integration fills. Therefore, it may be unclear which hierarchy you should open first to get a better view of the test results.
  • Both hierarchies can only have up to three levels.
  • Both hierarchies are exposed to users: users may add labels to them. Therefore, it's possible to define more than one label of a specific level (e.g., two suite labels). It's not clear how to deal with this. Currently, Allure will show it multiple times in the tree, which may be confusing.
  • It's harder to combine different integrations as they may provide different hierarchies out of the box.

Describe the solution you'd like
A solution is to introduce the default hierarchy directly to Allure model. The TestResult type should include a new property, say, titlePath, which, should be an array of strings (most probably).

Rules for integrations:

  1. An integration must translate the hierarchical structure of tests in the framework to titlePath values of the corresponding test results.
  2. An integration must respect all levels. Two test results have the same titlePath if and only if they
    a) correspond to the same test (e.g., are two retries of the test), or
    b) correspond to sibling tests (i.e., tests that share all their parents: suites/files/packages/etc).

Note

A combination of a test's titlePath and name can also be used to calculate the fullName value.

Allure 3 will use titlePath values to display the test results in a tree view similar to how it uses package labels. The main difference is that no splitting is involved (values are already arrays), which means no ambiguous split characters and no need for encoding/decoding.

Example

Suppose we have the following tests:

In foo.test.js:

describe("suite", () => {
  test("test-1", () => {});
  test("test-2", () => {});
});

In dir/bar.test.js:

describe("suite", () => {
  test("test-3", () => {});
});

In dir/baz.test.js:

describe("suite 1", () => {
  test("test-4", () => {});
});
describe("suite 2", () => {
  test("test-5", () => {});
});

When run, the test results with the following titlePath values are created:

["foo.test.js", "suite"] // for test-1 and test-2
["dir", "bar.test.js", "suite"] // for test-3
["dir", "baz.test.js", "suite 1"] // for test-4
["dir", "baz.test.js", "suite 2"] // for test-5

Allure Report will display the results in the form of the following tree (it may compress nodes with a sinlge child):

foo.test.js
├─ suite
│  ├─ test-1
│  ├─ test-2
dir
├─ bar.test.js
│  ├─ suite
│     ├─ test-3
├─ baz.test.js
   ├─ suite 1
   │  ├─ test-4
   ├─ suite 2
      ├─ test-5
@delatrie delatrie added theme:commons type:new feature New feature or request labels Feb 20, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
theme:commons type:new feature New feature or request
Projects
None yet
Development

No branches or pull requests

1 participant