diff --git a/docs/explanation/release_process/canary.rst b/docs/explanation/release_process/canary.rst index 314db29cd..a7c0d6403 100644 --- a/docs/explanation/release_process/canary.rst +++ b/docs/explanation/release_process/canary.rst @@ -93,4 +93,12 @@ The `beta` branch head is then updated on GitHub. Conclusion ---------- -This Canary Testing process, complete from snap building to testing, ensures that every release of Checkbox in the edge channel is thoroughly vetted and stable. \ No newline at end of file +This Canary Testing process, complete from snap building to testing, ensures that every release of Checkbox in the edge channel is thoroughly vetted and stable. + + +.. toctree:: + :hidden: + + canary_pipeline + validation_pipeline_execution + validation_job_example diff --git a/docs/tutorial/tutorial.rst b/docs/tutorial/tutorial.rst deleted file mode 100644 index 39142316b..000000000 --- a/docs/tutorial/tutorial.rst +++ /dev/null @@ -1,166 +0,0 @@ -:orphan: This document is built but not included in the table of content. - -.. _tutorials: - -Checkbox tutorials -================== - -.. note:: - If you're planning to develop tests for a Ubuntu Core system or - planning to package your tests as a Snap, see :doc:`/how-to/custom-app`. - -Creating an empty provider --------------------------- - -Checkbox Providers are bundles containing information how to run tests. - -To create an empty provider run:: - - $ checkbox-cli startprovider --empty com.example:myprovider - -``checkbox-cli`` is the command that runs Checkbox, ``--empty`` informs -Checkbox that you want to start from scratch. ``com.example:myprovider`` is the -name of the provider. Providers use IQN naming, it helps in tracking down -ownership of the provider. - -Checkbox Jobs are the things that describe how tests are run. Those Jobs are -defined in .pxu files, in 'units' directory of the provider. - -The provider we've just created doesn't have that directory, let's create it:: - - $ cd com.example\:myprovider - $ mkdir units - -Adding a simple job to a provider ---------------------------------- - -Jobs loosely follow RFC822 syntax. I.e. most content follow ``key:value`` -pattern. - -Let's add a simple job that runs a command. - -Open any ``.pxu`` file in ``units`` directory of the provider -(if there isn't any, just create one, like ``units.pxu``). -And add following content:: - - id: my-first-job - flags: simple - command: mycommand - -``id`` is used for identification purposes -``flags`` enables extra features. In the case of ``simple``, it lets us not -specify all the typical fields - Checkbox will infer some values for us. -``command`` specifies which command to run. Here it's ``mycommand`` - -In order for jobs to be visible in Checkbox they have to be included in some -test plan. Let's add a test plan definition to the same ``.pxu`` file.:: - - unit: test plan - id: first-tp - name: My first test plan - include: my-first-job - -.. warning:: - Separated entities in the .pxu file has to be separated by at least one - empty line. - - -Running jobs from a newly created provider ------------------------------------------- - -In order for Checkbox to `see` the provider we have to install it. -To do so run:: - - $ sudo ./manage.py install - -Now we're ready to launch Checkbox! Start the command line version with:: - - $ checkbox-cli - -Follow the instructions on the screen. The test will (probably) fail, because -of ``mycommand`` missing in your system. Let's change the job definition to do -something meaningful instead. Open ``units.pxu``, and change the line:: - - command: mycommand - -to :: - - command: [ `df -B 1G --output=avail $HOME |tail -n1` -gt 10 ] - -.. note:: - This command checks if there's at least 10GB of free space in $HOME - -This change won't be available just yet, as we still have an old version of the -provider installed in the system. Let's remove the previous version, and -install the new one.:: - - $ sudo rm -rf /usr/local/lib/plainbox-providers-1/com.example\:myprovider/ - $ sudo ./manage.py install - -This sudo operations (hopefully) look dangerous to you. See next part to see -how to avoid that. - -Developing provider without constantly reinstalling it ------------------------------------------------------- - -Instead of reinstalling the provider every time you change anything in it, you -can make Checkbox read it directly from the place you're changing it in.:: - - $ ./manage.py develop - -Because now Checkbox may see two instances of the same provider, make sure you -remove the previous one. - -.. note:: - ``./manage.py`` develop doesn't require sudo, as it makes all the - references in user's home. - -Improving job definition ------------------------- - -When you run Checkbox you see the job displayed as 'my-first-job' which is the -id of the job, which is not very human-friendly. This is because of the -``simple`` flag. Let's improve our Job definition. Open ``units.pxu`` and -replace the job definition with:: - - id: my-first-job - _summary: 10GB available in $HOME - _description: - this test checks if there's at least 10gb of free space in user's home - directory - plugin: shell - estimated_duration: 0.01 - command: [ `df -B 1G --output=avail $HOME |tail -n1` -gt 10 ] - -New stuff:: - - _summary: 10GB available in $HOME - -Summary is shown in Checkbox screens where jobs are selected. It's a -human-friendly identification of the job. It should should be short (50 - 70 -chars), as it's printed in one line. ``_`` means at the beginning means -the field is translatable. - -:: - - _purpose: - this test checks if there's at least 10gb of free space in user's home - directory - -Purpose as the name suggest should describe the purpose of the test. - - -:: - - plugin: shell - -Plugin tells Checkbox what kind of job is it. ``shell`` means it's a automated -test that runs a command and uses its return code to determine job's outcome. - -:: - - estimated_duration: 0.01 - -Tells Checkbox how long the test is expected to run. This field is currently -informative only. - diff --git a/docs/tutorial/using-checkbox/test-output.rst b/docs/tutorial/using-checkbox/test-output.rst deleted file mode 100644 index 18d18e930..000000000 --- a/docs/tutorial/using-checkbox/test-output.rst +++ /dev/null @@ -1,123 +0,0 @@ -:orphan: This document is built but not included in the table of content. - -.. _test_output: - -============================ -Test output -============================ - -When Checkbox runs the tests, it generates a series of outputs, which include -the ``ID`` and ``Category`` of the tests, along with the results of each of -them (if a test ID fails, the cause for the failure would be displayed). - -Now we will introduce each section of the test output by performing a test -for audio device detection in practice. - -.. code-block:: none - - =========[ Running job 1 / 3. Estimated time left (at least): 0:00:01 ]========= - -**Progress Indicator** shows the current status of the ongoing -job, including the total number of jobs and the number completed. It also -provides an estimated time left for completion, helping users track progress -and estimate when the test will finish. - - -.. code-block:: none - - -----------------------------[ Hardware Manifest ]------------------------------ - -**Summary** provides a brief sentence to help users understand the purpose of -this job. - -In this case, it's `Hardware Manifest -`_, which provides information -about various hardware devices used in the tests. When certain tests involve -specific hardware devices, Checkbox will prompt users to inform in advance -whether the machine includes specific hardware devices required for testing -purposes. - - -.. code-block:: none - - ID: com.canonical.plainbox::manifest - Category: com.canonical.plainbox::uncategorised - -**ID** indicates the job IDs run during the test. and it's organized into -different **Categories** based on their functionalities or characteristics. - - -.. code-block:: none - - ... 8< ------------------------------------------------------------------------- - ns: com.canonical.certification - name: checkbox-provider-base - has_audio_playback: False - has_audio_capture: True - - ------------------------------------------------------------------------- >8 --- - -This is the job stdout from ``com.canonical.plainbox::manifest``. If a job -fails or is skipped, the message will also be shown in this section. - -In our case, we are testing audio device detection, and the manifest was -prompted earlier to ask for user-provided information, this section displays -the collected information from the user. - - -.. code-block:: none - - Outcome: job passed - -At the end of each test, the **Outcome** will be displayed to indicate whether -the test has passed. - - -.. code-block:: none - - ------------[ Check that at least one audio playback device exits ]------------- - ID: com.canonical.certification::audio/detect-playback-devices - Category: com.canonical.plainbox::audio - Job cannot be started because: - - resource expression "manifest.has_audio_playback == 'True'" evaluates to false - Outcome: job cannot be started - -Oops! Apparently this job was skipped.Based on the stdout, it seems that this -issue occurred because when Checkbox asking hardware manifest, we mistakenly -set ``has_audio_playback`` to ``False``. Consequently, Checkbox determined that -the machine lacks the necessary audio devices to support the test. - - -.. code-block:: none - - ☑ : Hardware Manifest - ☐ : Check that at least one audio playback device exits - ☑ : Check that at least one audio capture device exists - -After all the jobs are completed, a checklist will summarize the results of -each test. - - -.. code-block:: none - - file:///home/user/.local/share/checkbox-ng/submission_2023-07-25T07.53.41.800141.html - file:///home/user/.local/share/checkbox-ng/submission_2023-07-25T07.53.41.800141.junit.xml - file:///home/user/.local/share/checkbox-ng/submission_2023-07-25T07.53.41.800141.tar.xz - -Checkbox collect all data related to the test and generates a **Test Reports** -locally. - - -.. code-block:: none - - Do you want to submit 'upload to certification' report? - y => yes - n => no - -Users would be asked whether they want to upload the results to the Canonical -:term:`Certification website`. - -Congrats! Now you have a basic understanding -of the test output. -