Collect system information (New) (#760)

* Init implenation of system info gathering

* System information collector init and vendorized inxi

* Store actual infos gathered by system_information in the state

* Make system_information in state persistent

* Include system_information in submission.json

* Tests for system_information collectors

* Metabox tests to check system_information

Minor: also port the fix (no reconnecting message) to inxi job

* Updated plainbox tests for new v8 state

* Update to overall class style

* Test class that resumes the session

* Avoid test_assistant's collection of system_information

* Test also build_SessionState

* Minor: small test for collect

* Make system_information executable

* Moved responsibility of system_information collection

Now the collection is done automatically upon the first get call
if no set was done previously. This should be effectively the same as
before, the first checkpoint dumps the information, reading it, while
a non-fresh session will load a checkpoint setting before reading it

Minor: Update tests to mock/check the new path

* Refactored Collectors to use metaclasses

Minor: moved return_code to output
Minor: moved success to property of CollectorOutput
Minor: moved str cast to actual cmd
Minor: updated tests

* Test also CollectorMeta

* Change REGISTER_NAME -> COLLECTOR_NAME

Minor: Better metaclass docstr

* Include inxi in the setup MANIFEST.in

Minor: exclude build dir from the installation

* Removed success from OutputSuccess and added super

* Mock inxi in metabox tests

* Removed additional system_information module

Minor: updated docstring in init to be up to date to the new content

* Documentation about System Information Collection

* Fixed paragraphs headers

* Address review comments

* Rename json_output -> payload

* Avoid building json in jinja

* Fix typing mistake and update main with the new impl.

* Versioned collector outputs

Minor: fixed typo

* Updated path system_information in vendor

* don't jsonify json

* Fixed loading mistyping the result of collection

checkbox-ng/MANIFEST.in

@@ -28,9 +28,11 @@ include plainbox/impl/providers/manifest/po/*.po
 include plainbox/impl/providers/manifest/po/*.pot
 include plainbox/impl/providers/manifest/po/POTFILES.in
 include plainbox/impl/providers/manifest/units/*.pxu
+include plainbox/vendor/inxi
 
 recursive-exclude daily-package-testing *
 recursive-include contrib *.policy
 recursive-include docs *.rst *.py *.html *.conf
 recursive-include plainbox/test-data *.json *.html *.tar.xz
 recursive-exclude debian *
+recursive-exclude build *

checkbox-ng/plainbox/impl/providers/exporters/data/checkbox.json

@@ -3,6 +3,7 @@
 {%- set resource_map = state.resource_map -%}
 {%- set job_state_map = state.job_state_map -%}
 {%- set category_map = state.category_map -%}
+{%- set system_information = state.system_information -%}
 {
     "title": {{ state.metadata.title | jsonify | safe }},
 {%- if "testplan_id" in app_blob %}
@@ -115,7 +116,8 @@
         {%- for cat_id, cat_name in category_map|dictsort %}
         "{{ cat_id }}": "{{ cat_name }}"{%- if not loop.last -%},{%- endif %}
         {%- endfor %}
-    }
+    },
+    "system_information": {{system_information.to_json() | safe}}
 {%- if ns ~ 'dkms_info_json' in state.job_state_map and state.job_state_map[ns ~ 'dkms_info_json'].result.outcome == 'pass' %},
 {%- set dkms_info_json = '{' + state.job_state_map[ns ~ 'dkms_info_json'].result.io_log_as_text_attachment.split('{', 1)[-1] %}
     "dkms_info": {{ dkms_info_json | indent(4, false) | safe }}

checkbox-ng/plainbox/impl/session/README.md

@@ -0,0 +1,79 @@
+# System Information Collection
+
+Checkbox collects various information about the system using Collectors.
+A Collector is a class that wraps a tool that can provide a JSON-serializable
+set of information.
+When Checkbox starts, it uses these collectors to collect and store
+in the session storage all the information it can gather. This is done before
+the first session checkpoint is created. From then onward, these information
+will remain in memory (and on disk in the checkpoint). When generating a
+submission report, Checkbox will include all the information in a top-level
+field of the json called "system_information".
+
+## Format
+
+A collector can either run succesfully or fail. Regardless of the result,
+running a collector will create a new field in the submission file following
+this format:
+
+```
+"system_information" : {
+  collector_name : {
+    "version" : collector_version,
+    "success" : true/false,
+    "outputs" : { ... }
+}
+```
+
+The outputs field's format depends on the success of the collection.
+
+If it ran successfully, the output field will have the follwing structure:
+
+```
+  "outputs" : {
+    "payload" : collector_parsed_json_output,
+    "stderr" : collector_error_log
+  }
+```
+Where the `collector.payload` is either an array or a dictionary.
+
+If it failed to run, the output field will have the following structure:
+
+```
+  "outputs" : {
+    "stdout" : collector_output_log,
+    "stderr" : collector_error_log
+  }
+```
+Where `collector_error_log` and `collector_output_log` are a string.
+
+## Creating new collectors
+
+To create a new collector, one has to create a class that uses the
+`CollectorMeta` metaclass. Additionally every collector has to define
+a `COLLECTOR_NAME`. Refer to the docstring of `CollectorMeta` for a more
+in-dept description.
+
+> Note: Before creating a new collector, verify if the functionality that
+> is needed is already implemented in an existing `collector`. If so, always
+> prefer using an already existing collector than creating a new one
+
+
+### Using external tools
+
+If the collector needs a tool, it should be added, when appropriate, to the
+vendorized section of Checkbox. Vendorization refers to the inclusion of
+external resource to a project's codebase.
+
+To vendorize a tool, locate the `vendor` module within Checkbox and place
+the vendorized version of the tool there, add to `vendor/__init__.py`
+the path to the executable that you have added.
+
+**It is appropriate** to add a vendorized tool when it is an executable script
+interpreted by any interpreter pre-installed on **every** version of Ubuntu that
+Checkbox supports. The tool must have a compatible license with the Checkbox
+license (GPLv3).
+
+**It is not appropriate** to add a compiled tool of any kind. Since Checkbox
+is designed to run on various architectures, compiled tools might not be
+universally compatible, leading to operational issues.

checkbox-ng/plainbox/impl/session/resume.py

@@ -63,6 +63,7 @@
 from plainbox.impl.secure.qualifiers import SimpleQualifier
 from plainbox.impl.session.state import SessionMetaData
 from plainbox.impl.session.state import SessionState
+from plainbox.impl.session.system_information import CollectionOutput, CollectorOutputs
 
 logger = logging.getLogger("plainbox.session.resume")
 
@@ -202,6 +203,8 @@ def _peek_json(self, json_repr):
             return SessionPeekHelper6().peek_json(json_repr)
         elif version == 7:
             return SessionPeekHelper7().peek_json(json_repr)
+        elif version == 8:
+            return SessionPeekHelper8().peek_json(json_repr)
         else:
             raise IncompatibleSessionError(
                 _("Unsupported version {}").format(version))
@@ -333,6 +336,9 @@ def _resume_json(self, json_repr, early_cb=None):
         elif version == 7:
             helper = SessionResumeHelper7(
                 self.job_list, self.flags, self.location)
+        elif version == 8:
+            helper = SessionResumeHelper8(
+                self.job_list, self.flags, self.location)
         else:
             raise IncompatibleSessionError(
                 _("Unsupported version {}").format(version))
@@ -586,6 +592,18 @@ class SessionPeekHelper7(MetaDataHelper7MixIn, SessionPeekHelper6):
     The only goal of this class is to reconstruct session state meta-data.
     """
 
+class SessionPeekHelper8(MetaDataHelper7MixIn, SessionPeekHelper6):
+    """
+    Helper class for implementing session peek feature
+
+    This class works with data constructed by
+    :class:`~plainbox.impl.session.suspend.SessionSuspendHelper7` which has
+    been pre-processed by :class:`SessionPeekHelper` (to strip the initial
+    envelope).
+
+    The only goal of this class is to reconstruct session state meta-data.
+    """
+
 class SessionResumeHelper1(MetaDataHelper1MixIn):
 
     """
@@ -1157,10 +1175,28 @@ def _build_SessionState(self, session_repr, early_cb=None):
         logger.debug(_("Resume complete!"))
         return session
 
-
 class SessionResumeHelper7(MetaDataHelper7MixIn, SessionResumeHelper6):
     pass
 
+class SessionResumeHelper8(SessionResumeHelper7):
+    def _restore_SessionState_system_information(self, session_state, session_repr):
+        _validate(session_repr, key="system_information", value_type=dict)
+        system_information = CollectorOutputs(
+            {
+                tool_name: CollectionOutput.from_dict(tool_output_json)
+                for (tool_name, tool_output_json) in session_repr[
+                    "system_information"
+                ].items()
+            }
+        )
+        session_state.system_information = system_information
+
+    def _build_SessionState(self, session_repr, early_cb=None):
+        session_state = super()._build_SessionState(session_repr, early_cb)
+        self._restore_SessionState_system_information(
+            session_state, session_repr
+        )
+        return session_state
 
 def _validate(obj, **flags):
     """Multi-purpose extraction and validation function."""

checkbox-ng/plainbox/impl/session/state.py

@@ -35,6 +35,9 @@
 from plainbox.impl.secure.qualifiers import select_jobs
 from plainbox.impl.session.jobs import JobState
 from plainbox.impl.session.jobs import UndesiredJobReadinessInhibitor
+from plainbox.impl.session.system_information import(
+    collect as collect_system_information
+)
 from plainbox.impl.unit.job import JobDefinition
 from plainbox.impl.unit.unit_with_id import UnitWithId
 from plainbox.impl.unit.testplan import TestPlanUnitSupport
@@ -746,6 +749,9 @@ def __init__(self, unit_list):
         self._resource_map = {}
         self._fake_resources = False
         self._metadata = SessionMetaData()
+        # If unset, this is loaded via system_information
+        self._system_information = None
+
         super(SessionState, self).__init__()
 
     def trim_job_list(self, qualifier):
@@ -815,6 +821,18 @@ def trim_job_list(self, qualifier):
                 self.on_job_removed(job)
                 self.on_unit_removed(job)
 
+    @property
+    def system_information(self):
+        if not self._system_information:
+            # This is a new session, we need to query this infos
+            self._system_information = collect_system_information()
+        return self._system_information
+
+    @system_information.setter
+    def system_information(self, value):
+        #TODO: check if system_information was already set
+        self._system_information = value
+
     def update_mandatory_job_list(self, mandatory_job_list):
         """
         Update the set of mandatory jobs (that must run).

checkbox-ng/plainbox/impl/session/suspend.py

@@ -661,6 +661,16 @@ def _repr_SessionMetaData(self, obj, session_dir):
         data['last_job_start_time'] = obj.last_job_start_time
         return data
 
+class SessionSuspendHelper8(SessionSuspendHelper7):
+    VERSION = 8
+
+    def _repr_SessionState(self, obj, session_dir):
+        data = super()._repr_SessionState(obj, session_dir)
+        data["system_information"] = {
+            tool_name: tool_output.to_dict()
+            for (tool_name, tool_output) in obj.system_information.items()
+        }
+        return data
 
 # Alias for the most recent version
-SessionSuspendHelper = SessionSuspendHelper7
+SessionSuspendHelper = SessionSuspendHelper8