Add 'expand' subcommand to list jobs and templates in a test plan (New)

[pieqq]

Description

The new subcommand "expands" a given test plan by going through all its sections and returning the pure (non-instantiated) jobs and templates being used.

It is different from the "list-boostrapped" subcommand since it does not perform any bootstrap.

By default, it outputs the jobs/templates id as text, but it can be called with the --format json option to output JSON data with all the known information for said jobs and templates.

This is to be used by external tools, for instance to create a document listing information (summary, description...) about executed jobs and templates.

Resolved issues

Fix CHECKBOX-1263

Documentation

Tests

Given the following units:

Test plans:

unit: test plan
id: demo-test-plan
name: demo-test-plan
bootstrap_include:
  demo_resource
  demo_resource_none
mandatory_include:
  demo-mandatory certification_status=non-blocker
nested_part:
  demo-nested1
include:
  demo-test-storages certification_status=blocker
  demo-test-storage.*
  demo-none-name
  .*demo-test-regex.*
  .*demo-none-regex.*
exclude:
  demo-job2
  demo-job-sibling4

unit: test plan
id: demo-nested1
name: demo-nested1
include:
  demo-job1 certification_status=blocker
  demo-job2
  demo-job-sibling1
  demo-job-sibling2
  demo-job-sibling3
  demo-job-sibling4

Resource jobs:

id: demo_resource
plugin: resource
command:
    echo 'name: sda'
    echo 'path: /dev/sda'
    echo
    echo 'name: sdb'
    echo 'path: /dev/sdb'

id: demo_resource_none
plugin: resource
command:
    echo ''

Template jobs:

unit: template
template-resource: demo_resource
template-id: demo-test-storages
template-summary: Demo the test storages
template-description:
  Very long description that explains what demoing the
  test storages means in the context of this change.
id: demo-test-storage-{name}
plugin: shell
command: echo "This is the path: {path}"
flags: also-after-suspend

# Used by a regex in the demo-test-plan
unit: template
template-resource: demo_resource
id: demo-test-regex-{name}
plugin: shell
command: echo "This is the path: {path}"
flags: also-after-suspend

unit: template
template-resource: demo_resource_none
id: demo-none-{name}
plugin: shell
command: echo "Resource always empty, should never show up: {path}"
flags: also-after-suspend

# Used by a regex in the demo-test-plan
unit: template
template-resource: demo_resource_none
id: demo-none-regex-{name}
plugin: shell
command: echo "Resource always empty, should never show up: {path}"
flags: also-after-suspend

Regular jobs:

id: demo-mandatory
plugin: shell
command: echo "This is mandatory job"
certification-status: blocker

id: demo-job1
plugin: shell
command: echo "This is job1"
flags: also-after-suspend

id: demo-job2
plugin: shell
command: echo "This is job2"
flags: also-after-suspend

id: demo-job-sibling1
plugin: shell
command: echo "This is a job"
siblings: [
  {"id": "demo-job-sibling2"},
  {"id": "demo-job-sibling3"},
  {"id": "demo-job-sibling4"}
  ]

We can run the expand subcommand like so:

checkbox-cli expand com.canonical.certification::demo-test-plan

Job 'com.canonical.certification::demo-mandatory'
Template 'com.canonical.certification::demo-test-storages'
Template 'com.canonical.certification::demo-none-name'
Template 'com.canonical.certification::demo-test-regex-name'
Template 'com.canonical.certification::demo-none-regex-name'
Job 'com.canonical.certification::demo-job1'
Job 'com.canonical.certification::demo-job-sibling1'
Job 'com.canonical.certification::demo-job-sibling2'
Job 'com.canonical.certification::demo-job-sibling3'

Observations:

• The list includes jobs in the mandatory_include section (here, demo-mandatory)
• All the jobs from the nested_part are included (demo-job1, demo-job-sibling1, demo-job-sibling[1-3])...
• ... except the ones that are also listed in the exclude section ( demo-job2, demo-job-sibling4)
• Templates that would not match any job are included (demo-none-name uses a resource job that does not return anything)
• Any job or template that matches any of the regular expressions are included (demo-test-regex-name, demo-none-regex-name)

Finally, the command can also output JSON data by adding --format json. Here is the output of checkbox-cli expand --format json com.canonical.certification::demo-test-plan | jq:

JSON output

[
  {
    "id": "com.canonical.certification::demo-mandatory",
    "plugin": "shell",
    "command": "echo \"This is mandatory job\"",
    "certification-status": "non-blocker",
    "unit": "job"
  },
  {
    "unit": "template",
    "template-resource": "demo_resource",
    "template-id": "com.canonical.certification::demo-test-storages",
    "template-summary": "Demo the test storages",
    "template-description": " Very long description that explains what demoing the\n test storages means in the context of this change.",
    "id": "com.canonical.certification::demo-test-storage-{name}",
    "plugin": "shell",
    "command": "echo \"This is the path: {path}\"",
    "flags": "also-after-suspend",
    "certification-status": "blocker"
  },
  {
    "unit": "template",
    "template-resource": "demo_resource_none",
    "id": "com.canonical.certification::demo-none-{name}",
    "plugin": "shell",
    "command": "echo \"Resource always empty, should never show up: {path}\"",
    "flags": "also-after-suspend",
    "certification-status": "unspecified",
    "template-id": "com.canonical.certification::demo-none-name"
  },
  {
    "unit": "template",
    "template-resource": "demo_resource",
    "id": "com.canonical.certification::demo-test-regex-{name}",
    "plugin": "shell",
    "command": "echo \"This is the path: {path}\"",
    "flags": "also-after-suspend",
    "certification-status": "unspecified",
    "template-id": "com.canonical.certification::demo-test-regex-name"
  },
  {
    "unit": "template",
    "template-resource": "demo_resource_none",
    "id": "com.canonical.certification::demo-none-regex-{name}",
    "plugin": "shell",
    "command": "echo \"Resource always empty, should never show up: {path}\"",
    "flags": "also-after-suspend",
    "certification-status": "unspecified",
    "template-id": "com.canonical.certification::demo-none-regex-name"
  },
  {
    "id": "com.canonical.certification::demo-job1",
    "plugin": "shell",
    "command": "echo \"This is job1\"",
    "flags": "also-after-suspend",
    "unit": "job",
    "certification-status": "blocker"
  },
  {
    "id": "com.canonical.certification::demo-job-sibling1",
    "plugin": "shell",
    "command": "echo \"This is a job\"",
    "siblings": "[\n {\"id\": \"demo-job-sibling2\"},\n {\"id\": \"demo-job-sibling3\"},\n {\"id\": \"demo-job-sibling4\"}\n ]",
    "unit": "job",
    "certification-status": "unspecified"
  },
  {
    "id": "com.canonical.certification::demo-job-sibling2",
    "plugin": "shell",
    "command": "echo \"This is a job\"",
    "unit": "job",
    "certification-status": "unspecified"
  },
  {
    "id": "com.canonical.certification::demo-job-sibling3",
    "plugin": "shell",
    "command": "echo \"This is a job\"",
    "unit": "job",
    "certification-status": "unspecified"
  }
]

[codecov]

Codecov Report

Attention: Patch coverage is 95.50562% with 4 lines in your changes are missing coverage. Please review.

Project coverage is 41.13%. Comparing base (539ec18) to head (8fa2195).
Report is 21 commits behind head on main.

Files	Patch %	Lines
checkbox-ng/checkbox_ng/launcher/subcommands.py	96.36%	0 Missing and 2 partials ⚠️
checkbox-ng/plainbox/impl/session/assistant.py	71.42%	2 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1065      +/-   ##
==========================================
+ Coverage   40.92%   41.13%   +0.21%     
==========================================
  Files         337      337              
  Lines       37547    37606      +59     
  Branches     6383     6398      +15     
==========================================
+ Hits        15365    15471     +106     
+ Misses      21543    21491      -52     
- Partials      639      644       +5     

Flag	Coverage Δ
checkbox-ng	67.41% <95.50%> (+0.46%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[tang-mm]

This works very well, thanks!

Adding a note here for future reference:
unlike list-bootstrapped, this command does not include the bootstrap section (bootstrap_include is ignored by default), so information-gathering jobs are not returned.

[Hook25]

This is ready to land imo. See if you like my suggestions, ty for changing the nomenclature (jobs -> unit) everywhere, it is clearer now!