From 020d7ef93f5ab7f583c000a7f5466af36cf7d358 Mon Sep 17 00:00:00 2001 From: Hook25 Date: Wed, 26 Jul 2023 14:19:34 +0200 Subject: [PATCH] Rebase to main and remove changes from temp branch --- docs/tutorial/base_tutorial.rst | 117 -------------------- docs/tutorial/extended_tutorial.rst | 141 ------------------------ docs/tutorial/index.rst | 13 +-- docs/tutorial/tutorial.rst | 164 ++++++++++++++++++++++++++++ 4 files changed, 168 insertions(+), 267 deletions(-) delete mode 100644 docs/tutorial/base_tutorial.rst delete mode 100644 docs/tutorial/extended_tutorial.rst create mode 100644 docs/tutorial/tutorial.rst diff --git a/docs/tutorial/base_tutorial.rst b/docs/tutorial/base_tutorial.rst deleted file mode 100644 index e41d95554..000000000 --- a/docs/tutorial/base_tutorial.rst +++ /dev/null @@ -1,117 +0,0 @@ -.. _tutorials: - -============= -Base Tutorial -============= - - This uses a new tutorial TP, its a small addition but could make a - huge difference, we can custom tailor a new user experience with it. - It will be referenced as tutorial-plan - -.. - - This uses explain and introduce interchangeably, by explain I don’t - mean a 300 pages novella about the feature but a dry simple - explanation of its purpose and how to use it # Introduction Here - mention what checkbox is - -Installing checkbox -------------------- - -Here we introduce snaps and debs for checkbox, introduce here the -concept of ``core`` and ``frontend``. - -Outcome: checkbox.checkbox-cli or checkbox-cli starting - -Running First Test Plan ------------------------ - -Here we can explain a little bit the different kind of jobs that we have -while we encounter them in the execution, explain comments and outcomes. - -Running a job will also ask if you want to submit it, it may be worth -explaining what that means! Also: outcome is saved, point to where and -how - -Outcome: tutorial-plan results - -Using basic launcher --------------------- - -~ ``checkbox-cli launcher.conf`` Outcome: Auto-starting tutorial-plan -test, Filtered test list - -Using basic configuration -------------------------- - -Explain how configs are used, **where to put them** Outcome: -Auto-starting tutorial-plan test via conf, filtered test list, also: -``check-config`` usage for config validation/see what is going on - -Machine Manifest ----------------- - -Explain how the machine manifest works/is created and where it is. -Create one with the create manifest job, now re launch and edit it. Edit -it via text editor. - -Using them all --------------- - -Here explain how conf/MM/launcher are prioritized between them (launcher -wins) and make the user do a couple of examples like setting start. - -Recall here ``check-config``, and show how it tracks the origin of -values - -Outcomes: Auto starting tutorial-plan via conf and launcher, -``check-config`` source explained - -Remote testing -============== - -Here say a couple of words on the motivation of remote testing, say that -if run with no args, checkbox-cli is local (and deprecated? useful to -check local configs? idk) - -Basic ------ - -Explicitly start two ends, one per terminal, to play around with them. - -Outcome: Two terminals, one running agent, one running controller, -tutorial-plan tests run “remotely” - -Configuration -------------- - -Explicitly explain how configuration interact with agent/controller, -with examples go through who reads what how, recall ``check-config`` and -show it works remotely - -Outcome: A couple of combination of configurations with examples, -running tests remotely with filters etc. - -Test Output -=========== - -Here guide the user through an output, use the tutorial plan output. - -Outcome: Understanding of the main, most common fields of the output - - Once here, the reader of the tutorial should have a good knowledge of - the basics of checkbox, he should be able to run tests, comprehend - the output and see/modify/debug the configuration - -Advanced Configs -================ - -Here explain every feature of config files, recall ``check-config`` -somewhere so that if someone skims through the guide it is known. - -This page is a pretty good beginning: -https://canonical-checkbox.readthedocs-hosted.com/en/stable/reference/launcher.html - -The only thing to change is to prune uncommon config values and -explanation and have an actionable config, per section, that the user -can load and “experience”. diff --git a/docs/tutorial/extended_tutorial.rst b/docs/tutorial/extended_tutorial.rst deleted file mode 100644 index c321ea1fa..000000000 --- a/docs/tutorial/extended_tutorial.rst +++ /dev/null @@ -1,141 +0,0 @@ -.. _adv_tutorials: - -================= -Extended Tutorial -================= - -Hacking basics -============== - -Link here to the QA tutorial for “normal users” and recall that we offer -snaps and debs. The -`CONTRIBUTING.md `__ -guide is a good basic as it explains how to install, sideload and launch -both remotely and locally, how to create new providers etc.. - - Basic -> Only use the most common options, think of ``name`` not - ``category-overrides`` - -Basic Test Plan -=============== - -Create a simple test plan that includes a few test cases from tutorial -or smoke, for example, to get the hang of it. - -Recall here side-loading. - -Outcome: Run the created test plan, show output - -Basic Test Case -=============== - -Now that we know how to create a custom test plan, lets create a custom -test case. - -Auto Test Case --------------- - -Introduce here the basic concepts of what a test case and write a simple -one. As a command do not use anything complicated (``true`` may be too -simple, but something easy like that, do not overwhelm the reader, here -he has to learn test cases, not linux). - -Link back to previous to say how to include this in a test plan. - -Outcome: Run the new test plan, show outcome, also create a failing one -and check that it does fail - -Manual Test Case ----------------- - -Introduce here the basics of the purpose/idea of a manual test case. - -Outcome: Run the test plan, show how the new test case is presented and -check that pass/fail is reported correctly - --------------- - - From here onward, we introduce features and use them in a test - case/plan, so that after each chapter it is clear what they do, how - to use them and how to use what they produce in practice - -Category -======== - -Outcome: Create a category, assign it to a test case - -Resources -========= - -Here mention what a resource is and name spacing, redirect to reference -for more. - -Basic usage ------------ - -Here introduce how resources are formatted and write a simple resource -job. This can be: is a file there? - -Now write a test that uses this resource (command true?), show that it -passes, remove the file, show that the job is skipped, introduce here -``fail-on-resource``. Show that it fails. - -Outcome: Test job that uses resources, understands how to create simple -ones and use them - -Combining resources -------------------- - -Here create a second resource job, this can be is another file present? - -Now write a test that uses one, one that uses the other and one that -uses both (``and``/``or``). Show how they are skipped or run for the -given resource evaluation results. - - Consider pointing to the complexity note in the reference here - -Ourcome: Two resource jobs, three tests, knowledge on how resource -expressions work. - -Manifest -======== - -Introduce what a manifest entity is and recall where the disk cache is -stored. - -Here we should only explain how to define a manifest and introduce the -basic kws (point to reference for more), then create a test plan that -uses one of the manifest entities that we just defined and run it, -filling the manifest as prompted. - -Mention here the job to show all manifest entities - -Outcome: Test that uses a manifest entity, knowledge on where the -manifest is store/how to read it - -Templates -========= - -Here introduce templates, point to reference for more. - -Basic Template --------------- - -The basic example -`here `__ -is pretty ok, I would simplify it a bit by removing one resource (3 -echos) and the filtering (``requires``) as it is explained above. Also, -less text, more doing. Command should be just a plain ``echo`` so that -the user can familiarize with the template substitution - -Outcome: Template that the user can try out and see the jobs that are -generated/run them and see a comprehensible output - -Jinja Templates ---------------- - -Explain here how to use jinja templates (``template-engine: jinja2``), -and provide a basic usage. Create a template that generates a job using -at least one jinja feature, let the user run the generated test plan. - -Outcome: Template that generates a test plan via jinja diff --git a/docs/tutorial/index.rst b/docs/tutorial/index.rst index 4807e32bb..344507085 100644 --- a/docs/tutorial/index.rst +++ b/docs/tutorial/index.rst @@ -20,24 +20,19 @@ problems, we want to help you, so please let us know - `get support <#>`_. Core tutorial ------------- -Follow the base tutorial steps in sequence; they take you on a learning +Follow the core tutorial steps in sequence; they take you on a learning journey through the product. .. toctree:: :maxdepth: 1 - base_tutorial + tutorial Extended tutorial ----------------- -Once you have completed the base tutorial, the extended optional tutorial +Once you have completed the core tutorial, the extended optional tutorial sections can be followed in any order - they don’t depend on each other. -Follow the base tutorial steps in sequence; they take you on a learning journey +Follow the core tutorial steps in sequence; they take you on a learning journey through the product. - -.. toctree:: - :maxdepth: 1 - - extended_tutorial diff --git a/docs/tutorial/tutorial.rst b/docs/tutorial/tutorial.rst new file mode 100644 index 000000000..e91ce1ce7 --- /dev/null +++ b/docs/tutorial/tutorial.rst @@ -0,0 +1,164 @@ +.. _tutorials: + +Checkbox tutorials +================== + +.. note:: + If you're planning to develop tests for a Ubuntu Core system or + planning to package your tests as a Snap, see :doc:`/how-to/custom-app`. + +Creating an empty provider +-------------------------- + +Checkbox Providers are bundles containing information how to run tests. + +To create an empty provider run:: + + $ checkbox-cli startprovider --empty com.example:myprovider + +``checkbox-cli`` is the command that runs Checkbox, ``--empty`` informs +Checkbox that you want to start from scratch. ``com.example:myprovider`` is the +name of the provider. Providers use IQN naming, it helps in tracking down +ownership of the provider. + +Checkbox Jobs are the things that describe how tests are run. Those Jobs are +defined in .pxu files, in 'units' directory of the provider. + +The provider we've just created doesn't have that directory, let's create it:: + + $ cd com.example\:myprovider + $ mkdir units + +Adding a simple job to a provider +--------------------------------- + +Jobs loosely follow RFC822 syntax. I.e. most content follow ``key:value`` +pattern. + +Let's add a simple job that runs a command. + +Open any ``.pxu`` file in ``units`` directory of the provider +(if there isn't any, just create one, like ``units.pxu``). +And add following content:: + + id: my-first-job + flags: simple + command: mycommand + +``id`` is used for identification purposes +``flags`` enables extra features. In the case of ``simple``, it lets us not +specify all the typical fields - Checkbox will infer some values for us. +``command`` specifies which command to run. Here it's ``mycommand`` + +In order for jobs to be visible in Checkbox they have to be included in some +test plan. Let's add a test plan definition to the same ``.pxu`` file.:: + + unit: test plan + id: first-tp + name: My first test plan + include: my-first-job + +.. warning:: + Separated entities in the .pxu file has to be separated by at least one + empty line. + + +Running jobs from a newly created provider +------------------------------------------ + +In order for Checkbox to `see` the provider we have to install it. +To do so run:: + + $ sudo ./manage.py install + +Now we're ready to launch Checkbox! Start the command line version with:: + + $ checkbox-cli + +Follow the instructions on the screen. The test will (probably) fail, because +of ``mycommand`` missing in your system. Let's change the job definition to do +something meaningful instead. Open ``units.pxu``, and change the line:: + + command: mycommand + +to :: + + command: [ `df -B 1G --output=avail $HOME |tail -n1` -gt 10 ] + +.. note:: + This command checks if there's at least 10GB of free space in $HOME + +This change won't be available just yet, as we still have an old version of the +provider installed in the system. Let's remove the previous version, and +install the new one.:: + + $ sudo rm -rf /usr/local/lib/plainbox-providers-1/com.example\:myprovider/ + $ sudo ./manage.py install + +This sudo operations (hopefully) look dangerous to you. See next part to see +how to avoid that. + +Developing provider without constantly reinstalling it +------------------------------------------------------ + +Instead of reinstalling the provider every time you change anything in it, you +can make Checkbox read it directly from the place you're changing it in.:: + + $ ./manage.py develop + +Because now Checkbox may see two instances of the same provider, make sure you +remove the previous one. + +.. note:: + ``./manage.py`` develop doesn't require sudo, as it makes all the + references in user's home. + +Improving job definition +------------------------ + +When you run Checkbox you see the job displayed as 'my-first-job' which is the +id of the job, which is not very human-friendly. This is because of the +``simple`` flag. Let's improve our Job definition. Open ``units.pxu`` and +replace the job definition with:: + + id: my-first-job + _summary: 10GB available in $HOME + _description: + this test checks if there's at least 10gb of free space in user's home + directory + plugin: shell + estimated_duration: 0.01 + command: [ `df -B 1G --output=avail $HOME |tail -n1` -gt 10 ] + +New stuff:: + + _summary: 10GB available in $HOME + +Summary is shown in Checkbox screens where jobs are selected. It's a +human-friendly identification of the job. It should should be short (50 - 70 +chars), as it's printed in one line. ``_`` means at the beginning means +the field is translatable. + +:: + + _purpose: + this test checks if there's at least 10gb of free space in user's home + directory + +Purpose as the name suggest should describe the purpose of the test. + + +:: + + plugin: shell + +Plugin tells Checkbox what kind of job is it. ``shell`` means it's a automated +test that runs a command and uses its return code to determine job's outcome. + +:: + + estimated_duration: 0.01 + +Tells Checkbox how long the test is expected to run. This field is currently +informative only. +