-
Notifications
You must be signed in to change notification settings - Fork 49
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Add snap refresh/revert jobs to SRU test plans and add manifest entri…
…es to control when to execute them (New) (#883) Fix CHECKBOX-718 * Add manifest entries to test gadget/kernel/snapd snaps The snap-refresh-*-from-[stable|base]-rev jobs are executed if the revision of the related snap (gadget, kernel, snapd) currently installed on the device is different from the revision in base version and/or stable channel. It is better to provide an option to decide whether or not these tests should be executed, per device. This is achieved using three manifest entries (one per snap type). If these entries are explicitely set to True in the manifest, they will be executed (if they match the condition described above, of course). Otherwise, they will be skipped. * Move snap refresh/revert jobs and test plans in the snapd section These jobs and test plans are not specific to Ubuntu Core and can be run on any device running snapd (classic, server, core and hybrid images). This commit will: - Move jobs and test plans from the the base provider's ubuntucore/ section to the snapd/ section - Adjust their names (rename prefixes from ubuntucore to snapd) - Adjust their categories (from ubuntucore to snapd) * Add snap-refresh-revert nested part to SRU and snappy-snap-automated Doing so ensure the snap refresh/revert tests are executed on every device running the SRU test plan (mostly classic images) as well as on devices running IoT-related test plans (mostly Ubuntu core images). * Add README.md for the base provider's snapd section * Move the snap refresh/revert manifest to the snapd section * Add some explanation in snapd manifest file
- Loading branch information
Showing
7 changed files
with
347 additions
and
267 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,52 @@ | ||
| # Snap Refresh/Revert Tests | ||
|
|
||
| ## Rationale | ||
|
|
||
| Snapd offers the ability to refresh a given snap to a specific revision, | ||
| and to revert to the previously installed one if needs be. | ||
|
|
||
| Moreover, Snapd has mechanisms in place to automatically refresh every | ||
| snap to the latest revision available in their tracked channel. This is | ||
| interesting for devices in the field, but also to automatically update a | ||
| system that has just been setup, for instance. | ||
|
|
||
| In order to test these features, the `snapd` section of Checkbox base | ||
| provider offers a test plan, `snap-refresh-revert`, that contains jobs doing | ||
| the following: | ||
|
|
||
| 1. Generate resource information for any gadget, kernel or snapd snap available | ||
| on the system; then, if the currently installed revision is different from | ||
| the targeted revision and if required (see "Manifest entries" section below): | ||
| 2. Refresh them again, but this time to their base revision (the revision | ||
| that came pre-installed with the image) | ||
| 3. Revert them back to their original revision | ||
| 4. Refresh each of these snaps to the revision number from the stable channel | ||
| 5. Revert them back | ||
|
|
||
| After each of these steps, the device is rebooted, then Checkbox checks if | ||
| the given snap has been updated to the expected revision before proceeding | ||
| the next step. | ||
|
|
||
| ## Manifest entries | ||
|
|
||
| For each of the snap types (gadget, kernel, snapd), a manifest entry controls | ||
| whether or not the tests should be executed. If the manifest entries are | ||
| not defined, Checkbox will skip every job in the test plan by default. To | ||
| enable these tests, the following manifest entries can be set to `true`: | ||
|
|
||
| - `need_gadget_snap_update_test` | ||
| - `need_kernel_snap_update_test` | ||
| - `need_snapd_snap_update_test` | ||
|
|
||
| See the [Manifest Entry] section of Checkbox documentation for more | ||
| information. | ||
|
|
||
| This test plan is aimed at being used during the Snap Update Verification | ||
| (SUV) process or the Stable Release Update (SRU) process. | ||
|
|
||
| During SUV in particular, the snap that is installed before running the | ||
| test plan is from the beta (or the candidate) channel, so it should always | ||
| be different from the version in the stable channel or the base version | ||
| present on the system. | ||
|
|
||
| [Manifest Entry]: https://checkbox.readthedocs.io/en/stable/reference/units/manifest-entry.html |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,207 @@ | ||
| id: snap_revision_info | ||
| plugin: resource | ||
| _summary: | ||
| Gather name, tracking, and revision info of the kernel, snapd and gadget snaps | ||
| estimated_duration: 3s | ||
| category_id: snapd | ||
| command: | ||
| snap_update_test.py --resource | ||
|
|
||
| unit: template | ||
| template-resource: snap_revision_info | ||
| template-unit: job | ||
| id: snapd/snap-refresh-{type}-{name}-to-stable-rev | ||
| _summary: Refresh {name} snap to latest revision in stable channel | ||
| _description: | ||
| This test is currently for SUV process, the snap to be tested is on the beta | ||
| channel. Test the availability to refresh to the older version on the stable | ||
| channel. This test will be excuted only when the current installed revision | ||
| (on beta channel) is different from the revsion in stable channel. | ||
| plugin: shell | ||
| estimated_duration: 1m | ||
| category_id: snapd | ||
| user: root | ||
| imports: | ||
| from com.canonical.certification import snap_revision_info | ||
| from com.canonical.plainbox import manifest | ||
| requires: | ||
| (snap_revision_info.name == "{name}") and snap_revision_info.stable_rev != snap_revision_info.original_installed_rev | ||
| manifest.need_{type}_snap_update_test == "True" | ||
| command: | ||
| path="$PLAINBOX_SESSION_SHARE/{name}_snap_revision_info" | ||
| snap_update_test.py --refresh --revision {stable_rev} --info-path "$path" {name} | ||
|
|
||
| unit: template | ||
| template-resource: snap_revision_info | ||
| template-unit: job | ||
| id: snapd/reboot-after-snap-refresh-{type}-{name}-to-stable-rev | ||
| _summary: Reboot after {name} snap refresh to latest revision in stable channel | ||
| plugin: shell | ||
| flags: noreturn autorestart | ||
| estimated_duration: 3m | ||
| category_id: snapd | ||
| user: root | ||
| depends: snapd/snap-refresh-{type}-{name}-to-stable-rev | ||
| command: | ||
| echo "Waiting 90s for any snap operation to finish before rebooting..." | ||
| sleep 90 | ||
| reboot | ||
|
|
||
| unit: template | ||
| template-resource: snap_revision_info | ||
| template-unit: job | ||
| id: snapd/snap-verify-after-refresh-{type}-{name}-to-stable-rev | ||
| _summary: Verify {name} snap revision after refreshing to latest revision in stable channel | ||
| plugin: shell | ||
| estimated_duration: 30s | ||
| category_id: snapd | ||
| user: root | ||
| depends: snapd/reboot-after-snap-refresh-{type}-{name}-to-stable-rev | ||
| command: | ||
| path="$PLAINBOX_SESSION_SHARE/{name}_snap_revision_info" | ||
| snap_update_test.py --verify-refresh --info-path "$path" {name} | ||
|
|
||
| unit: template | ||
| template-resource: snap_revision_info | ||
| template-unit: job | ||
| id: snapd/snap-revert-{type}-{name}-from-stable-rev | ||
| _summary: Revert {name} snap to original revision from stable channel | ||
| plugin: shell | ||
| estimated_duration: 3m | ||
| category_id: snapd | ||
| user: root | ||
| depends: snapd/snap-verify-after-refresh-{type}-{name}-to-stable-rev | ||
| command: | ||
| path="$PLAINBOX_SESSION_SHARE/{name}_snap_revision_info" | ||
| snap_update_test.py --revert --info-path "$path" {name} | ||
|
|
||
| unit: template | ||
| template-resource: snap_revision_info | ||
| template-unit: job | ||
| id: snapd/reboot-after-snap-revert-{type}-{name}-from-stable-rev | ||
| _summary: Reboot after {name} snap reverting to latest revision in stable channel | ||
| plugin: shell | ||
| flags: noreturn autorestart | ||
| estimated_duration: 3m | ||
| category_id: snapd | ||
| user: root | ||
| depends: snapd/snap-revert-{type}-{name}-from-stable-rev | ||
| command: | ||
| echo "Waiting 90s for any snap operation to finish before rebooting..." | ||
| sleep 90 | ||
| reboot | ||
|
|
||
| unit: template | ||
| template-resource: snap_revision_info | ||
| template-unit: job | ||
| id: snapd/snap-verify-after-revert-{type}-{name}-from-stable-rev | ||
| _summary: Verify {name} snap revision after reverting from stable revision | ||
| plugin: shell | ||
| estimated_duration: 3s | ||
| category_id: snapd | ||
| user: root | ||
| depends: snapd/reboot-after-snap-revert-{type}-{name}-from-stable-rev | ||
| command: | ||
| path="$PLAINBOX_SESSION_SHARE/{name}_snap_revision_info" | ||
| snap_update_test.py --verify-revert --info-path "$path" {name} | ||
|
|
||
| unit: template | ||
| template-resource: snap_revision_info | ||
| template-unit: job | ||
| id: snapd/snap-refresh-{type}-{name}-to-base-rev | ||
| _summary: Refresh {name} snap to its base revision | ||
| _description: | ||
| This test is currently for SUV process, the snap to be tested is on the beta | ||
| channel. Test the availability to refresh to the base revision (the revision | ||
| that came with the image). This test will be excuted only when the current | ||
| installed revision (on beta channel) is different from the base revision. | ||
| This job (and the jobs depending on it) are useful when the device cannot | ||
| be reprovisioned easily, but you still want to simulate a fresh install | ||
| that would bump to the latest revision available. | ||
| plugin: shell | ||
| estimated_duration: 3m | ||
| category_id: snapd | ||
| user: root | ||
| imports: | ||
| from com.canonical.certification import snap_revision_info | ||
| from com.canonical.plainbox import manifest | ||
| requires: | ||
| (snap_revision_info.name == "{name}") and snap_revision_info.base_rev != snap_revision_info.original_installed_rev | ||
| manifest.need_{type}_snap_update_test == "True" | ||
| command: | ||
| path="$PLAINBOX_SESSION_SHARE/{name}_snap_revision_info" | ||
| snap_update_test.py --refresh --revision {base_rev} --info-path "$path" {name} | ||
|
|
||
| unit: template | ||
| template-resource: snap_revision_info | ||
| template-unit: job | ||
| id: snapd/reboot-after-snap-refresh-{type}-{name}-to-base-rev | ||
| _summary: Reboot after {name} snap refresh to base revision | ||
| plugin: shell | ||
| flags: noreturn autorestart | ||
| estimated_duration: 3m | ||
| category_id: snapd | ||
| user: root | ||
| depends: snapd/snap-refresh-{type}-{name}-to-base-rev | ||
| command: | ||
| echo "Waiting 90s for any snap operation to finish before rebooting..." | ||
| sleep 90 | ||
| reboot | ||
|
|
||
| unit: template | ||
| template-resource: snap_revision_info | ||
| template-unit: job | ||
| id: snapd/snap-verify-after-refresh-{type}-{name}-to-base-rev | ||
| _summary: Verify {name} snap revision after refreshing to base revision | ||
| plugin: shell | ||
| estimated_duration: 30s | ||
| category_id: snapd | ||
| user: root | ||
| depends: snapd/reboot-after-snap-refresh-{type}-{name}-to-base-rev | ||
| command: | ||
| path="$PLAINBOX_SESSION_SHARE/{name}_snap_revision_info" | ||
| snap_update_test.py --verify-refresh --info-path "$path" {name} | ||
|
|
||
| unit: template | ||
| template-resource: snap_revision_info | ||
| template-unit: job | ||
| id: snapd/snap-revert-{type}-{name}-from-base-rev | ||
| _summary: Revert {name} snap from base revision to original revision | ||
| plugin: shell | ||
| estimated_duration: 3m | ||
| category_id: snapd | ||
| user: root | ||
| depends: snapd/snap-verify-after-refresh-{type}-{name}-to-base-rev | ||
| command: | ||
| path="$PLAINBOX_SESSION_SHARE/{name}_snap_revision_info" | ||
| snap_update_test.py --revert --info-path "$path" {name} | ||
|
|
||
| unit: template | ||
| template-resource: snap_revision_info | ||
| template-unit: job | ||
| id: snapd/reboot-after-snap-revert-{type}-{name}-from-base-rev | ||
| _summary: Reboot after {name} snap revert to base revision | ||
| plugin: shell | ||
| flags: noreturn autorestart | ||
| estimated_duration: 3m | ||
| category_id: snapd | ||
| user: root | ||
| depends: snapd/snap-revert-{type}-{name}-from-base-rev | ||
| command: | ||
| echo "Waiting 90s for any snap operation to finish before rebooting..." | ||
| sleep 90 | ||
| reboot | ||
|
|
||
| unit: template | ||
| template-resource: snap_revision_info | ||
| template-unit: job | ||
| id: snapd/snap-verify-after-revert-{type}-{name}-from-base-rev | ||
| _summary: Verify {name} snap revision after reverting from base revision | ||
| plugin: shell | ||
| estimated_duration: 3s | ||
| category_id: snapd | ||
| user: root | ||
| depends: snapd/reboot-after-snap-revert-{type}-{name}-from-base-rev | ||
| command: | ||
| path="$PLAINBOX_SESSION_SHARE/{name}_snap_revision_info" | ||
| snap_update_test.py --verify-revert --info-path "$path" {name} |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,22 @@ | ||
|
|
||
| # If the following manifest entries are set to true, SRU and Cert IoT test | ||
| # plans will try to run snap refresh/revert to and from the base revision and | ||
| # the stable revision. See README.md for more information. | ||
|
|
||
| unit: manifest entry | ||
| id: need_kernel_snap_update_test | ||
| _prompt: Do the following snaps types require snap refresh/revert testing? | ||
| _name: Kernel | ||
| value-type: bool | ||
|
|
||
| unit: manifest entry | ||
| id: need_snapd_snap_update_test | ||
| _prompt: Do the following snaps types require snap refresh/revert testing? | ||
| _name: Snapd | ||
| value-type: bool | ||
|
|
||
| unit: manifest entry | ||
| id: need_gadget_snap_update_test | ||
| _prompt: Do the following snaps types require snap refresh/revert testing? | ||
| _name: Gadget | ||
| value-type: bool |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.