Retire pm_test from PC suspend 30 cycles and separate IoT suspend cyc…

…les (New) (canonical#601) * Retire pm_test from PC suspend 30 cycles and saperate IoT suspend cycles * Remove the useless flag preserve-local * Make suspend (fwts and rtcwake) flow more linear * Correct the summary of resources jobs * Modify the summary. * Remove unnecessary variables * Seperate the suspend_cycles_reboot test case to a new file * Add the detail description in md file. * Fix the typo, and add the short description at the beginning of suspend_cycles_reboot.pxu * Break lines of text at 80 characters as possible as I can * Fix some tiny problems
LiaoU3 · Mar 20, 2024 · 3a0a45a · 3a0a45a
1 parent c688f8c
commit 3a0a45a
Show file tree

Hide file tree

Showing 7 changed files with 341 additions and 3 deletions.
diff --git a/providers/base/bin/suspend.sh b/providers/base/bin/suspend.sh
@@ -0,0 +1,14 @@
+#!/bin/bash
+# FWTS s3 only support x86_64 and i386 cpu
+# ref: https://github.com/ColinIanKing/fwts/blob/master/src/acpi/s3/s3.c#L24
+# So we need to seperate two kind of situation (FWTS_S3 supported or not)
+
+architecture=$(uname -m)
+if [ "$architecture" = "x86_64" ] || [ "$architecture" = "i386" ]; then
+    export LD_LIBRARY_PATH="$LD_LIBRARY_PATH:$SNAP/usr/lib/fwts"
+    set -o pipefail
+    checkbox-support-fwts_test -f none -s s3 --s3-device-check --s3-device-check-delay="${STRESS_S3_WAIT_DELAY:-45}" --s3-sleep-delay="${STRESS_S3_SLEEP_DELAY:-30}"
+else
+    rtcwake -v -m mem -s "${STRESS_S3_SLEEP_DELAY:-30}"
+fi
+
diff --git a/providers/base/units/stress/s3s4.pxu b/providers/base/units/stress/s3s4.pxu
@@ -150,3 +150,4 @@ requires: cpuinfo.platform in ("i386", "x86_64")
 command: [ -e "$PLAINBOX_SESSION_SHARE"/hibernate_{s4_iterations}_cycles.log ] && cat "$PLAINBOX_SESSION_SHARE"/hibernate_{s4_iterations}_cycles.log
 _description:
  Attaches the log from the {s4_iterations} cycles Hibernate/Resume test if it exists
+
diff --git a/providers/base/units/stress/suspend_cycles_reboot.md b/providers/base/units/stress/suspend_cycles_reboot.md
@@ -0,0 +1,106 @@
+# Flow of the suspend-cycles-stress-test test plan
+
+This description will focus on the suspend cycles and reboot process.
+
+The remaining work log check, suspend time check, and log attachments will be
+executed at the end of a suspend and reboot jobs.
+
+## Definition of the test case name
+
+- **suspend\_cycles\_{n}\_reboot{k} :**
+  - Indicates the execution of a suspend operation, n is the suspend index of\
+    the k<sup>th</sup> round of reboot.
+  - For example: S<sub>k</sub>n
+- **suspend\_cycles\_reboot{k}:**
+  - Indicate the execution of a reboot operation, k is the the reboot index.
+  - For example: R<sub>k</sub>
+
+## Example
+
+If doing 5 suspends per reboot round for 3 rounds (N = 5, K = 3), it means:
+
+- `n`: numbers of suspend  in each reboot
+- `k`: numbers of reboot
+
+- suspend\_cycles\_1\_reboot1:
+  - S<sub>A</sub>1
+- suspend\_cycles_1\_reboot{{suspend\_reboot\_id}}:
+  - S<sub>k</sub>1 (`k`: from A to C)
+- suspend\_cycles\_{{suspend\_id}}\_reboot{{suspend\_reboot\_id}}:
+  - S<sub>k</sub>n (`n`: from 2 to 5, `k`: from A to C)
+- suspend\_cycles\_reboot{{suspend\_reboot\_id}}:
+  - R<sub>k</sub> (`k`: from A to C)
+
+The flow will be the following:
+
+S<sub>A</sub>1 &rarr; S<sub>A</sub>2 &rarr; S<sub>A</sub>3 &rarr; S<sub>A</sub>4 &rarr; S<sub>A</sub>5 &rarr; R<sub>A</sub>
+
+&rarr; S<sub>B</sub>1 &rarr; S<sub>B</sub>2 &rarr; S<sub>B</sub>3 &rarr; S<sub>B</sub>4 &rarr; S<sub>B</sub>5 &rarr; R<sub>B</sub>
+
+&rarr; S<sub>C</sub>1 &rarr; S<sub>C</sub>2 &rarr; S<sub>C</sub>3 &rarr; S<sub>C</sub>4 &rarr; S<sub>C</sub>5 &rarr; R<sub>C</sub>
+
+## Relation between template and resource jobs
+
+- suspend\_cycles\_1\_reboot1: job
+  - For example: S<sub>A</sub>1
+- suspend\_cycles\_1\_reboot{2...k}: template job
+  - For example: S<sub>B</sub>1, S<sub>C</sub>1
+  - After job:
+    - suspend\_cycles\_reboot{{suspend\_reboot\_previous}}
+      - For example: R<sub>A</sub>,  R<sub>B</sub>
+  - Resource job:
+    - stress\_s3\_cycles\_iterations\_1
+      - Output:
+        - suspend\_reboot\_id: reboot index
+          - For example: B, C
+        - suspend\_reboot\_previous: previous reboot index
+          - For example: A, B
+- suspend\_cycles\_{2…n}\_reboot{1...k}: template job
+  - For example:
+    - S<sub>A</sub>2, S<sub>A</sub>3, S<sub>A</sub>4, S<sub>A</sub>5
+    - S<sub>B</sub>2, S<sub>B</sub>3, S<sub>B</sub>4, S<sub>B</sub>5
+    - S<sub>C</sub>2, S<sub>C</sub>3, S<sub>C</sub>4, S<sub>C</sub>5
+  - After job:
+    - suspend\_cycles\_{{suspend\_id\_previous}}\_reboot{{suspend\_reboot\_id}}
+      - For example:
+        - S<sub>A</sub>1, S<sub>A</sub>2, S<sub>A</sub>3, S<sub>A</sub>4
+        - S<sub>B</sub>1, S<sub>B</sub>2, S<sub>B</sub>3, S<sub>B</sub>4
+        - S<sub>B</sub>1, S<sub>C</sub>2, S<sub>C</sub>3, S<sub>C</sub>4
+  - Resource job:
+    - stress\_s3\_cycles\_iterations\_multiple
+      - Output:
+        - suspend\_id: suspend index
+          - For example: 2, 3, 4, 5
+        - suspend\_id\_previous: previous suspend index
+          - For example: 1, 2, 3, 4
+        - suspend\_reboot\_id: reboot index
+          - For example: A, B, C
+- suspend\_cycles\_reboot{1...k}: template job
+  - For example: R<sub>A</sub>, R<sub>B</sub>, R<sub>C</sub>
+  - After job:
+    - suspend\_cycles\_{{s3\_iterations}}\_reboot{{suspend\_reboot\_id}}
+      - For example: S<sub>A</sub>5, S<sub>B</sub>5, S<sub>C</sub>5
+  - Resource job:
+    - stress\_suspend\_reboot\_cycles\_iterations
+      - Output:
+        - s3\_iterations: numbers of suspend  in each reboo
+          - For example: 5
+        - suspend\_reboot\_id: reboot index
+          - For example: A, B, C
+
+Or, as a table:
+
+| Name of Job or Template Job | S<sub>A</sub>1 |          S<sub>k</sub>1           |                                                S<sub>k</sub>n                                                 |                 R<sub>k</sub>                  |
+| --------------------------- |:--------------:|:---------------------------------:|:-------------------------------------------------------------------------------------------------------------:|:----------------------------------------------:|
+| Resource Job               |      None      | stress\_s3\_cycles\_iterations\_1 |                                   stress\_s3\_cycles\_iterations\_multiple                                    |  stress\_suspend\_reboot\_cycles\_iterations   |
+| Generated Job               | S<sub>A</sub>1 |  S<sub>B</sub>1, S<sub>C</sub>1   | S<sub>A</sub>2, ..., S<sub>A</sub>5; S<sub>B</sub>2, ..., S<sub>B</sub>5; S<sub>C</sub>2, ..., S<sub>C</sub>5 |  R<sub>A</sub>, R<sub>B</sub>, R<sub>C</sub>   |
+| After Job                   |      None      |   R<sub>A</sub>,  R<sub>B</sub>   | S<sub>A</sub>1, ..., S<sub>A</sub>4; S<sub>B</sub>1, ..., S<sub>B</sub>4; S<sub>C</sub>1, ..., S<sub>C</sub>4 | S<sub>A</sub>5, S<sub>B</sub>5, S<sub>C</sub>5 |
+
+### Test case link flow
+
+|    S<sub>A</sub>1  & S<sub>k</sub>1    |                                     S<sub>k</sub>n                                      |    R<sub>k</sub>     |
+|:--------------------------------------:|:---------------------------------------------------------------------------------------:|:--------------------:|
+|             S<sub>A</sub>1             | &rarr; S<sub>A</sub>2 &rarr; S<sub>A</sub>3 &rarr; S<sub>A</sub>4 &rarr; S<sub>A</sub>5 | &rarr; R<sub>A</sub> |
+| ( R<sub>A</sub> )&rarr; S<sub>B</sub>1 | &rarr; S<sub>B</sub>2 &rarr; S<sub>B</sub>3 &rarr; S<sub>B</sub>4 &rarr; S<sub>B</sub>5 | &rarr; R<sub>B</sub> |
+| ( R<sub>B</sub> )&rarr; S<sub>C</sub>1 | &rarr; S<sub>C</sub>2 &rarr; S<sub>C</sub>3 &rarr; S<sub>C</sub>4 &rarr; S<sub>C</sub>5 | &rarr; R<sub>C</sub> |
+<!-- markdownlint-enbale MD033 -->
diff --git a/providers/base/units/stress/suspend_cycles_reboot.pxu b/providers/base/units/stress/suspend_cycles_reboot.pxu
@@ -0,0 +1,184 @@
+# This set of test cases is used to perform the suspend `n` cycles for `k` reboot
+# cycles.
+# For more details, please view the suspend_cycles_reboot.md file in this
+# folder.
+
+id: stress_s3_cycles_iterations_1
+category_id: stress-tests/suspend
+summary:
+  This resource generates suspend_reboot iteration numbers for suspend_cycles_1_reboot{2..k} jobs
+plugin: resource
+environ: STRESS_SUSPEND_REBOOT_ITERATIONS
+command:
+ for i in $(seq 2 "${STRESS_SUSPEND_REBOOT_ITERATIONS:-3}"); do
+     echo "suspend_reboot_id: $i"
+     echo "suspend_reboot_previous: $((i-1))"
+     echo
+ done
+estimated_duration: 1s
+
+id: stress_s3_cycles_iterations_multiple
+category_id: stress-tests/suspend
+summary:
+  This resource generates suspend_reboot iteration numbers for suspend_cycles_{2..n}_reboot{1..k} jobs
+plugin: resource
+environ: STRESS_S3_ITERATIONS STRESS_SUSPEND_REBOOT_ITERATIONS
+command:
+ for i in $(seq 1 "${STRESS_SUSPEND_REBOOT_ITERATIONS:-3}"); do
+   for j in $(seq 2 "${STRESS_S3_ITERATIONS:-30}"); do
+     echo "suspend_id: $j"
+     echo "suspend_id_previous: $((j-1))"
+     echo "suspend_reboot_id: $i"
+     echo
+   done
+ done
+estimated_duration: 1s
+
+id: stress_suspend_reboot_cycles_iterations
+category_id: stress-tests/suspend
+summary:
+  This resource generates suspend_reboot iteration numbers for suspend_cycles_reboot{1..k} jobs
+plugin: resource
+environ: STRESS_S3_ITERATIONS STRESS_SUSPEND_REBOOT_ITERATIONS
+command:
+ for i in $(seq 1 "${STRESS_SUSPEND_REBOOT_ITERATIONS:-3}"); do
+   echo "s3_iterations: ${STRESS_S3_ITERATIONS:-30}"
+   echo "suspend_reboot_id: $i"
+   echo
+ done
+estimated_duration: 1s
+
+id: stress_suspend_reboot_iterations
+category_id: stress-tests/suspend
+summary:
+  This resource generates suspend_reboot iteration numbers for suspend-{n}-cycles-with-reboot-{k}-{log/time}-check jobs
+plugin: resource
+environ: STRESS_S3_ITERATIONS STRESS_SUSPEND_REBOOT_ITERATIONS STRESS_SUSPEND_SLEEP_THRESHOLD STRESS_SUSPEND_RESUME_THRESHOLD
+command:
+ echo "reboot_iterations: ${STRESS_SUSPEND_REBOOT_ITERATIONS:-3}"
+ echo "s3_iterations: ${STRESS_S3_ITERATIONS:-30}"
+ echo "resume_threshold: ${STRESS_SUSPEND_SLEEP_THRESHOLD:-10}"
+ echo "sleep_threshold: ${STRESS_SUSPEND_RESUME_THRESHOLD:-5}"
+ echo
+estimated_duration: 1s
+
+unit: job
+plugin: shell
+category_id: stress-tests/suspend
+id: stress-tests/suspend_cycles_1_reboot1
+imports:
+  from com.canonical.certification import sleep
+  from com.canonical.certification import rtc
+requires:
+  sleep.mem == 'supported'
+  rtc.state == 'supported'
+estimated_duration: 75.0
+environ: PLAINBOX_SESSION_SHARE STRESS_S3_SLEEP_DELAY STRESS_S3_WAIT_DELAY LD_LIBRARY_PATH
+user: root
+command:
+   suspend.sh 2>&1 | tee -a "$PLAINBOX_SESSION_SHARE"/suspend_cycles_with_reboot_total.log
+summary:
+    Suspend and resume device (suspend cycle 1, reboot cycle 1)
+
+unit: template
+template-resource: stress_s3_cycles_iterations_1
+template-unit: job
+template-engine: jinja2
+plugin: shell
+category_id: stress-tests/suspend
+id: stress-tests/suspend_cycles_1_reboot{{suspend_reboot_id}}
+imports:
+  from com.canonical.certification import sleep
+  from com.canonical.certification import rtc
+requires:
+  sleep.mem == 'supported'
+  rtc.state == 'supported'
+estimated_duration: 75.0
+environ: PLAINBOX_SESSION_SHARE STRESS_S3_SLEEP_DELAY STRESS_S3_WAIT_DELAY LD_LIBRARY_PATH
+after: stress-tests/suspend_cycles_reboot{{suspend_reboot_previous}}
+user: root
+command:
+ suspend.sh 2>&1 | tee -a "$PLAINBOX_SESSION_SHARE"/suspend_cycles_with_reboot_total.log
+summary:
+    Suspend and resume device (suspend cycle 1, reboot cycle {{suspend_reboot_id}})
+
+unit: template
+template-resource: stress_s3_cycles_iterations_multiple
+template-unit: job
+template-engine: jinja2
+plugin: shell
+category_id: stress-tests/suspend
+id: stress-tests/suspend_cycles_{{suspend_id}}_reboot{{suspend_reboot_id}}
+imports:
+  from com.canonical.certification import sleep
+  from com.canonical.certification import rtc
+requires:
+  sleep.mem == 'supported'
+  rtc.state == 'supported'
+estimated_duration: 75.0
+environ: PLAINBOX_SESSION_SHARE STRESS_S3_SLEEP_DELAY STRESS_S3_WAIT_DELAY LD_LIBRARY_PATH
+after: stress-tests/suspend_cycles_{{suspend_id_previous}}_reboot{{suspend_reboot_id}}
+user: root
+command:
+ suspend.sh 2>&1 | tee -a "$PLAINBOX_SESSION_SHARE"/suspend_cycles_with_reboot_total.log
+summary:
+    Suspend and resume device (suspend cycle {{suspend_id}}, reboot cycle {{suspend_reboot_id}})
+
+unit: template
+template-resource: stress_suspend_reboot_cycles_iterations
+template-unit: job
+template-engine: jinja2
+plugin: shell
+category_id: stress-tests/suspend
+id: stress-tests/suspend_cycles_reboot{{suspend_reboot_id}}
+imports:
+  from com.canonical.certification import sleep
+  from com.canonical.certification import rtc
+requires:
+  sleep.mem == 'supported'
+  rtc.state == 'supported'
+estimated_duration: 30.0
+user: root
+command:
+ reboot
+flags: noreturn autorestart
+after: stress-tests/suspend_cycles_{{s3_iterations}}_reboot{{suspend_reboot_id}}
+summary:
+   Reboot device (reboot cycle {{suspend_reboot_id}}).
+
+unit: template
+template-resource: stress_suspend_reboot_iterations
+template-unit: job
+plugin: shell
+category_id: stress-tests/suspend
+id: stress-tests/suspend-{s3_iterations}-cycles-with-reboot-{reboot_iterations}-log-check
+requires: cpuinfo.platform in ("i386", "x86_64")
+estimated_duration: 1.0
+command: [ -e "$PLAINBOX_SESSION_SHARE"/suspend_cycles_with_reboot_total.log ] && sleep_test_log_check.py -v --ignore-warning -t all "$PLAINBOX_SESSION_SHARE"/suspend_cycles_with_reboot_total.log
+after: stress-tests/suspend_cycles_reboot{reboot_iterations}
+summary:
+ Automated check of the {s3_iterations} cycles Suspend/Resume with reboot {reboot_iterations} suspend log for errors detected by fwts.
+
+unit: template
+template-resource: stress_suspend_reboot_iterations
+template-unit: job
+plugin: shell
+category_id: stress-tests/suspend
+id: stress-tests/suspend-{s3_iterations}-cycles-with-reboot-{reboot_iterations}-time-check
+requires: cpuinfo.platform in ("i386", "x86_64")
+estimated_duration: 1.0
+command: [ -e "$PLAINBOX_SESSION_SHARE"/suspend_cycles_with_reboot_total.log ] && sleep_time_check.py "$PLAINBOX_SESSION_SHARE"/suspend_cycles_with_reboot_total.log --s {sleep_threshold} --r {resume_threshold}
+after: stress-tests/suspend_cycles_reboot{reboot_iterations}
+summary:
+ Automated check of the {s3_iterations} cycles Suspend/Resume with reboot {reboot_iterations} suspend log for time.
+
+unit: template
+template-resource: stress_suspend_reboot_iterations
+template-unit: job
+plugin: attachment
+category_id: stress-tests/suspend
+id: stress-tests/suspend-{s3_iterations}-cycles-with-reboot-{reboot_iterations}-log-attach
+estimated_duration: 1.0
+command: [ -e "$PLAINBOX_SESSION_SHARE"/suspend_cycles_with_reboot_total.log ] && cat "$PLAINBOX_SESSION_SHARE"/suspend_cycles_with_reboot_total.log
+summary:
+ Attach the suspend {s3_iterations} cycles with reboot {reboot_iterations} total log.
diff --git a/providers/base/units/stress/test-plan.pxu b/providers/base/units/stress/test-plan.pxu
@@ -129,7 +129,7 @@ include:
 nested_part:
     warm-boot-stress-test
     cold-boot-stress-test
-    suspend-stress-test
+    suspend-cycles-stress-test
     hibernate-stress-test
     stress-ng-automated
 
@@ -251,6 +251,39 @@ mandatory_include:
     meminfo
     interface
 
+unit: test plan
+id: suspend-cycles-stress-test
+_name: Suspend (S3) cycles stress test
+_description:
+ Suspends the machine a pre-defined number of times and on
+ resume of OS performs a hardware check to ensure all
+ items are still present.
+estimated_duration: 42h
+bootstrap_include:
+  stress_s3_cycles_iterations_1
+  stress_s3_cycles_iterations_multiple
+  stress_suspend_reboot_cycles_iterations
+  stress_suspend_reboot_iterations
+include:
+  stress-tests/suspend_cycles.*
+  stress-tests/suspend-.*-cycles.*
+mandatory_include:
+    com.canonical.plainbox::manifest
+    package
+    snap
+    uname
+    lsb
+    cpuinfo
+    dpkg
+    dmi_attachment
+    sysfs_attachment
+    udev_attachment
+    lspci_attachment
+    lsusb_attachment
+    dmi
+    meminfo
+    interface
+
 id: stress-ng-automated
 unit: test plan
 _name: Automated stress-ng tests

diff --git a/providers/certification-client/units/client-cert-desktop-22-04.pxu b/providers/certification-client/units/client-cert-desktop-22-04.pxu
@@ -172,6 +172,6 @@ nested_part:
     stress-ng-cert-automated
     stress-iperf3-automated
     #stress-cert-full
-    stress-suspend-30-cycles-with-reboots-automated
+    suspend-cycles-stress-test
     stress-warmboot-coldboot-automated
     stress-pm-graph
diff --git a/providers/certification-client/units/client-cert-odm-desktop-22-04.pxu b/providers/certification-client/units/client-cert-odm-desktop-22-04.pxu
@@ -172,7 +172,7 @@ nested_part:
     stress-iperf3-automated
     warm-boot-stress-test
     cold-boot-stress-test
-    suspend-stress-test
+    suspend-cycles-stress-test
     stress-ng-cert-automated
     tpm-cert-automated
     info-attachment-cert-full