Change output format to report overview #840

matejmatuska · 2023-08-16T14:53:24Z

This patch improves the leapp output format. The "REPORT" section is
renamed to "REPORT OVERVIEW" and in addition to high and medium priority
report titles it contains also inhibitor titles and actor and short
error message for errors.

The long (possibly thousands of lines) detailed errors messaged and
tracebacks are kept in the "ERRORS" section. The inhibitors section is
removed, because it's contents are now in the report overview.

github-actions · 2023-08-16T14:53:38Z

Thank you for contributing to the Leapp project!

Please note that every PR needs to comply with the Leapp Guidelines and must pass all tests in order to be mergeable.
If you want to request a review or rebuild a package in copr, you can use following commands as a comment:

review please @oamg/developers to notify leapp developers of the review request
/packit copr-build to submit a public copr build using packit

To launch regression testing public members of oamg organization can leave the following comment:

/rerun to schedule basic regression tests using this pr build and leapp-repository*master* as artifacts
/rerun 42 to schedule basic regression tests using this pr build and leapp-repository*PR42* as artifacts
/rerun-sst to schedule sst tests using this pr build and leapp-repository*master* as artifacts
/rerun-sst 42 to schedule sst tests using this pr build and leapp-repository*PR42* as artifacts

Please open ticket in case you experience technical problem with the CI. (RH internal only)

Note: In case there are problems with tests not being triggered automatically on new PR/commit or pending for a long time, please consider rerunning the CI by commenting leapp-ci build (might require several comments). If the problem persists, contact leapp-infra.

matejmatuska · 2023-08-16T14:54:00Z

Requires #838

Leapp introduced new output apis (and removed some other). See oamg/leapp#840 for details.

matejmatuska · 2023-08-16T15:19:24Z

Examples

Normal

Inhibitor

Errors (couldn't fit into one image)

NOTE: The "Stable Key report entry not provided, dynamically generating one ..." messages are only visible with --debug.

pirat89 · 2023-08-17T11:51:23Z

/rerun 1116

github-actions · 2023-08-17T12:06:44Z

Copr build succeeded: https://copr.fedorainfracloud.org/coprs/build/6311627

github-actions · 2023-08-17T12:11:52Z

Copr build succeeded: https://copr.fedorainfracloud.org/coprs/build/6311680

github-actions · 2023-08-17T13:37:46Z

Testing Farm request for RHEL-8.6-rhui/6311680;6311627 regression testing has been created.
Once finished, results should be available here.
Full pipeline log.

github-actions · 2023-08-17T13:48:04Z

Testing Farm request for RHEL-7.9-rhui/6311680;6311627 regression testing has been created.
Once finished, results should be available here.
Full pipeline log.

dkubek

lgtm! The code looks good, did not find any major issues nor obvious bugs. I ran upgrade and preupgrade and simulated different inhibitors and errors and the output looks at advertised

pirat89

I need to do some testing yet, as it's kinda late, I am possibly overlook something and my comments could be wrong. Let's sync in the morning how we will handle this.

packaging/leapp.spec

pirat89 · 2023-08-22T18:59:35Z

leapp/reporting/__init__.py

        # Before removing Flags, the original inhibitor detection worked
        # by checking the `flags` field; keep the `flags` field until we drop Flags completely
        # Currently we know that 'flags' does not exist otherwise so it's
        # safe to just set it
        report['flags'] = [Groups.INHIBITOR]
+    if Groups.ERROR in report.get('groups', []):
+        # see above for explanation
+        report['flags'] = [Groups.ERROR]


@matejmatuska @dkubek @fernflower I've realized this is problematic, as the list of flags is specified in the report-schema-v110.json. so the json file cannot contain the flag. to save a time, when report v110 is generated, ignore the error flag (in the way it's not present in the json file). it's compatible however with v120, as groups are not enumerated.
we could introduce report v111, however, I am afraid we cannot set it as default anyway as I am not sure how it will be accepted by plugins for cockpit, insights, and satellite/foreman.

@fernflower wdyt? could we do a fast check in the morning whether all mentioned plugins are ok with the new flag, so we could introduce report v111 and set it as default?

also, we could keep error as a tag, but it does not fit there well. let's discuss that in the morning

thinking about that after the sleep, I vote to go the most safe way. We will drop it from the json file completely and it will be covered properly the next time.

In this case it shows up as in the 1.2.0 schema:

"groups": [ "error" ],

And not flags/tags field.

This should be the problematic line:
https://github.com/oamg/leapp/pull/840/files#diff-a0dcff5a59a744244ee3056a16cd7fca4bbcc6cd573c6d6e9dd26ddb4205723eL151

leapp/reporting/__init__.py

Leapp introduced new output apis (and removed some other). See oamg/leapp#840 for details. The calls to `generate_report_file()` in `commands/upgrade/utils.py` are swapped to workaround a bug in the framework where the json report generation modifies messages passed to it. Jira: OAMG-9663

Currently there is no straightforward way to identify whether a report is an ordinary report or an error report (report generated from an error). An `ERROR` group is introduced in `leapp.reporting.Groups.ERROR` which is used to group error reports. This is utilized in the leapp output to show the number of error reports in the summary. Also error reports in `leapp-report.txt` are now followed by a "(error)" string similar to inhibitors. For now ERROR isn't in flags nor tags in the leapp-report.json in report schema 1.1.0. This because integration with plugins might break, due to the enumeration of possible flags in the schema. This will be adjusted later. In report schema 1.2.0 the ERROR group is listed in groups as expected. Jira: OAMG-9532

This patch improves the leapp output format. The "REPORT" section is renamed to "REPORT OVERVIEW" and in addition to high and medium priority report titles it contains also inhibitor titles and actor and short error message for errors. The long (possibly thousands of lines) detailed errors messaged and tracebacks are kept in the "ERRORS" section. The inhibitors section is removed, because it's contents are now in the report overview. Jira: OAMG-9663

pirat89 · 2023-08-23T13:53:51Z

Tested manually, works as expected, see below.

error

============================================================
                           ERRORS                           
============================================================

2023-08-23 14:59:34.971322 [ERROR] Actor: pes_events_scanner
Message: Data file /etc/leapp/files/pes-events.json is invalid or could not be retrieved.
Summary:
    Details: Could not fetch pes-events.json from https://cert.cloud.redhat.com/api/pes/2/pes-events.json (error code: 401).
    Hint: Read documentation at: https://access.redhat.com/articles/3664871 for more information about how to retrieve the file.

============================================================
                       END OF ERRORS                        
============================================================

2023-08-23 14:59:35.109 WARNING  PID: 17795 leapp.reporting: Stable Key report entry not provided, dynamically generating one - 93b26ca870d07eabd42892a70ed8a91556d9b74c
Debug output written to /var/log/leapp/leapp-upgrade.log
2023-08-23 14:59:35.140 WARNING  PID: 17795 leapp.reporting: Stable Key report entry not provided, dynamically generating one - 93b26ca870d07eabd42892a70ed8a91556d9b74c

============================================================
                      REPORT OVERVIEW                       
============================================================

Following errors occurred and the upgrade cannot continue:
    1. Actor: pes_events_scanner
       Message: Data file /etc/leapp/files/pes-events.json is invalid or could not be retrieved.

Reports summary:
    Errors:                      1
    Inhibitors:                  0
    HIGH severity reports:       0
    MEDIUM severity reports:     0
    LOW severity reports:        0
    INFO severity reports:       1

Before continuing consult the full report:
    A report has been generated at /var/log/leapp/leapp-report.json
    A report has been generated at /var/log/leapp/leapp-report.txt

============================================================
                   END OF REPORT OVERVIEW                   
============================================================

============================================================
                      REPORT OVERVIEW                       
============================================================

Following errors occurred and the upgrade cannot continue:
    1. Actor: dnf_transaction_check
       Message: DNF execution failed with non zero exit code.

HIGH and MEDIUM severity reports:
    1. Packages available in excluded repositories will not be installed
    2. GRUB2 core will be automatically updated during the upgrade
    3. Packages not signed by Red Hat found on the system
    4. Difference in Python versions and support in RHEL 8
    5. Upgrade is unsupported
    6. Module pam_pkcs11 will be removed from PAM configuration
    7. chrony using default configuration

Reports summary:
    Errors:                      1
    Inhibitors:                  0
    HIGH severity reports:       5
    MEDIUM severity reports:     2
    LOW severity reports:        4
    INFO severity reports:       4

Before continuing consult the full report:
    A report has been generated at /var/log/leapp/leapp-report.json
    A report has been generated at /var/log/leapp/leapp-report.txt

# grep "(error)" -A1 /var/log/leapp/leapp-report.txt
Risk Factor: high (error)
Title: DNF execution failed with non zero exit code.

## snippet from json file:
   {
      "title": "DNF execution failed with non zero exit code.",
      "timeStamp": "2023-08-23T13:21:15.150183Z",
      "hostname": "localhost.localdomain",
      "actor": "dnf_transaction_check",
      "summary" ....,
      "audience": "sysadmin", 
      "flags": [],
      "key": "1e26c8b86cec72c25c04ca045d2b0ed3e8ca2df5",
      "id": "e7c19634f1ab2e6887c69ea627b0ffc8d17dabbded26d5904934a81ef97ee26d",
      "tags": [],
      "severity": "high"
    }

For the schema v1.2.0:

============================================================
                      REPORT OVERVIEW                       
============================================================

Following errors occurred and the upgrade cannot continue:
    1. Actor: dnf_transaction_check
       Message: DNF execution failed with non zero exit code.

HIGH and MEDIUM severity reports:
    1. Packages available in excluded repositories will not be installed
    2. GRUB2 core will be automatically updated during the upgrade
    3. Difference in Python versions and support in RHEL 8
    4. Packages not signed by Red Hat found on the system
    5. Upgrade is unsupported
    6. Module pam_pkcs11 will be removed from PAM configuration
    7. chrony using default configuration

Reports summary:
    Errors:                      1
    Inhibitors:                  0
    HIGH severity reports:       5
    MEDIUM severity reports:     2
    LOW severity reports:        4
    INFO severity reports:       4

Before continuing consult the full report:
    A report has been generated at /var/log/leapp/leapp-report.json
    A report has been generated at /var/log/leapp/leapp-report.txt

============================================================
                   END OF REPORT OVERVIEW                   
============================================================

# grep "(error)" -A1 /var/log/leapp/leapp-report.txt
Risk Factor: high (error)
Title: DNF execution failed with non zero exit code.

Snippet from the json file

   {
      "groups": [
        "error"
      ],
      "title": "DNF execution failed with non zero exit code.",
      "timeStamp": "2023-08-23T13:40:04.309613Z",
      "hostname": "localhost.localdomain",
      "actor": "dnf_transaction_check",

The rest of entries looks good as I expected too. I forgot to put here inhibitors (will try to get outputs also later) but it seemed correct to me also. Here is one positive:

============================================================
                      REPORT OVERVIEW                       
============================================================

HIGH and MEDIUM severity reports:
    1. Packages available in excluded repositories will not be installed
    2. Upgrade is unsupported
    3. GRUB2 core will be automatically updated during the upgrade
    4. Difference in Python versions and support in RHEL 8
    5. Packages not signed by Red Hat found on the system
    6. Module pam_pkcs11 will be removed from PAM configuration
    7. chrony using default configuration

Reports summary:
    Errors:                      0
    Inhibitors:                  0
    HIGH severity reports:       5
    MEDIUM severity reports:     2
    LOW severity reports:        4
    INFO severity reports:       3

Before continuing consult the full report:
    A report has been generated at /var/log/leapp/leapp-report.json
    A report has been generated at /var/log/leapp/leapp-report.txt

============================================================
                   END OF REPORT OVERVIEW                   
============================================================

pirat89

see my previous comments. works as expected. merging

Leapp introduced new output apis (and removed some other). See oamg/leapp#840 for details. The calls to `generate_report_file()` in `commands/upgrade/utils.py` are swapped to workaround a bug in the framework where the json report generation modifies messages passed to it. Jira: OAMG-9663

## Packaging - Provide leapp-framework 5.0 (oamg#818, oamg#840) ## Framework ### Fixes - Improve processing of reports with UTF-8 characters (oamg#821, oamg#828) - Fix info reporting with only one path to log (oamg#834) ### Enhancements - Include tracebacks of actors into the leapp.db (oamg#832) ## Leapp (tool) ### Fixes - Dialog creation fails not more for a component without choices (oamg#826) - Empty report is generated correctly (oamg#828) - Fix processing data in remediation instructions with non-ascii characters () ### Enhancements - Improve report summary output to make it more visible (oamg#818, oamg#840) ## stdlib ### Fixes - Fixed the call when the execute cannot be performed (oamg#836) ### Enhancements - changes related just to stdlib - under leapp/libraries/stdlib

## Packaging - Provide leapp-framework 5.0 (#818, #840) ## Framework ### Fixes - Improve processing of reports with UTF-8 characters (#821, #828) - Fix info reporting with only one path to log (#834) ### Enhancements - Include tracebacks of actors into the leapp.db (#832) ## Leapp (tool) ### Fixes - Dialog creation fails not more for a component without choices (#826) - Empty report is generated correctly (#828) - Fix processing data in remediation instructions with non-ascii characters () ### Enhancements - Improve report summary output to make it more visible (#818, #840) ## stdlib ### Fixes - Fixed the call when the execute cannot be performed (#836) ### Enhancements - changes related just to stdlib - under leapp/libraries/stdlib

matejmatuska added a commit to matejmatuska/leapp-repository that referenced this pull request Aug 16, 2023

Use new leapp output apis

d317a29

Leapp introduced new output apis (and removed some other). See oamg/leapp#840 for details.

matejmatuska mentioned this pull request Aug 16, 2023

Use new leapp output apis oamg/leapp-repository#1116

Merged

pirat89 added this to the 7.9/8.9/9.3 milestone Aug 17, 2023

pirat89 added the enhancement label Aug 17, 2023

matejmatuska mentioned this pull request Aug 21, 2023

Add an ERROR report group #838

Closed

matejmatuska force-pushed the report-merge-poc branch from e640bcb to 0fb569e Compare August 21, 2023 11:33

dkubek self-assigned this Aug 21, 2023

dkubek previously approved these changes Aug 21, 2023

View reviewed changes

pirat89 requested changes Aug 22, 2023

View reviewed changes

pirat89 reviewed Aug 22, 2023

View reviewed changes

leapp/reporting/__init__.py Outdated Show resolved Hide resolved

pirat89 self-assigned this Aug 23, 2023

matejmatuska dismissed dkubek’s stale review via b30dba1 August 23, 2023 08:56

pirat89 added the Requries Repo PR Use it when leapp-deps is changed label Aug 23, 2023

matejmatuska force-pushed the report-merge-poc branch from 84c7dea to 6cdb2c3 Compare August 23, 2023 11:41

matejmatuska added 2 commits August 23, 2023 15:28

matejmatuska force-pushed the report-merge-poc branch from 6cdb2c3 to 4b0d48c Compare August 23, 2023 13:28

pirat89 approved these changes Aug 23, 2023

View reviewed changes

pirat89 merged commit 2bfda2b into oamg:master Aug 23, 2023
14 checks passed

pirat89 added the changelog-checked The merger/reviewer checked the changelog draft document and updated it when relevant label Aug 23, 2023

pirat89 mentioned this pull request Aug 23, 2023

Release 0.16.0 #841

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Change output format to report overview #840

Change output format to report overview #840

matejmatuska commented Aug 16, 2023

github-actions bot commented Aug 16, 2023

matejmatuska commented Aug 16, 2023

matejmatuska commented Aug 16, 2023

pirat89 commented Aug 17, 2023

github-actions bot commented Aug 17, 2023

github-actions bot commented Aug 17, 2023

github-actions bot commented Aug 17, 2023

github-actions bot commented Aug 17, 2023

dkubek left a comment

pirat89 left a comment

pirat89 Aug 22, 2023 •

edited

Loading

pirat89 Aug 22, 2023

pirat89 Aug 23, 2023

matejmatuska Aug 23, 2023 •

edited

Loading

pirat89 commented Aug 23, 2023 •

edited

Loading

pirat89 left a comment

Change output format to report overview #840

Change output format to report overview #840

Conversation

matejmatuska commented Aug 16, 2023

github-actions bot commented Aug 16, 2023

Thank you for contributing to the Leapp project!

matejmatuska commented Aug 16, 2023

matejmatuska commented Aug 16, 2023

Examples

pirat89 commented Aug 17, 2023

github-actions bot commented Aug 17, 2023

github-actions bot commented Aug 17, 2023

github-actions bot commented Aug 17, 2023

github-actions bot commented Aug 17, 2023

dkubek left a comment

Choose a reason for hiding this comment

pirat89 left a comment

Choose a reason for hiding this comment

pirat89 Aug 22, 2023 • edited Loading

Choose a reason for hiding this comment

pirat89 Aug 22, 2023

Choose a reason for hiding this comment

pirat89 Aug 23, 2023

Choose a reason for hiding this comment

matejmatuska Aug 23, 2023 • edited Loading

Choose a reason for hiding this comment

pirat89 commented Aug 23, 2023 • edited Loading

error

pirat89 left a comment

Choose a reason for hiding this comment

pirat89 Aug 22, 2023 •

edited

Loading

matejmatuska Aug 23, 2023 •

edited

Loading

pirat89 commented Aug 23, 2023 •

edited

Loading