Add a custom_junit callback plugin #132

mgirgisf · 2024-07-22T07:24:54Z

This custom_junit plugin extends the existing junit plugin and produces an XML report with classes and names that polarian expects.

softwarefactory-project-zuul · 2024-07-22T08:06:34Z

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://review.rdoproject.org/zuul/buildset/e8c007fe0bc8493baff88e377f733728

✔️ openstack-k8s-operators-content-provider SUCCESS in 40m 47s
❌ functional-tests-on-osp18 FAILURE in 25m 00s

mgirgisf · 2024-07-23T05:51:29Z

recheck

softwarefactory-project-zuul · 2024-07-23T07:14:34Z

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://review.rdoproject.org/zuul/buildset/c8c7ff17afe04310af8bb8e233af99a1

✔️ openstack-k8s-operators-content-provider SUCCESS in 1h 21m 47s
❌ functional-tests-on-osp18 FAILURE in 1h 05m 20s

elfiesmelfie · 2024-07-23T15:13:32Z

roles/telemetry_autoscaling/tasks/configure_heat.yml

+    - name: Add failure to test result
+      delegate_to: localhost
+      lineinfile:
+        path: "{{ logs_dir }}/test_run_result.xml"


The buildset logs are showing that this file doesn't exist: https://logserver.rdoproject.org/32/132/8c02c82c6426d08d090b9ae8a036610ab59f5c6f/github-check/functional-tests-on-osp18/3ccb604/controller/ci-framework-data/logs/post_deploy_run_functional_test.log

The logs at https://logserver.rdoproject.org/32/132/8c02c82c6426d08d090b9ae8a036610ab59f5c6f/github-check/functional-tests-on-osp18/3ccb604/controller/ci-framework-data/logs/functional-tests/
have an autoscaling_results file.

You need to update the file names to match.

softwarefactory-project-zuul · 2024-07-24T07:20:17Z

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://review.rdoproject.org/zuul/buildset/46d9b206bd794ec2b8a5752e4e7080f4

✔️ openstack-k8s-operators-content-provider SUCCESS in 1h 30m 52s
❌ functional-tests-on-osp18 FAILURE in 1h 16m 14s

softwarefactory-project-zuul · 2024-07-25T11:53:03Z

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://softwarefactory-project.io/zuul/t/rdoproject.org/buildset/49968e7a68554e05a78f3a2266edf975

✔️ openstack-k8s-operators-content-provider SUCCESS in 1h 35m 47s
❌ functional-tests-on-osp18 FAILURE in 1h 06m 47s

softwarefactory-project-zuul · 2024-07-25T14:16:06Z

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://softwarefactory-project.io/zuul/t/rdoproject.org/buildset/3c4a444c4e1d40a0b1dcbd3cd550bb96

✔️ openstack-k8s-operators-content-provider SUCCESS in 1h 51m 13s
❌ functional-tests-on-osp18 FAILURE in 1h 06m 22s

softwarefactory-project-zuul · 2024-08-01T13:14:25Z

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://softwarefactory-project.io/zuul/t/rdoproject.org/buildset/80c5859d0993422db465a39200fc2aeb

✔️ feature-verification-tests-noop SUCCESS in 5s
✔️ openstack-k8s-operators-content-provider SUCCESS in 1h 39m 12s
❌ functional-tests-on-osp18 FAILURE in 1h 23m 50s

elfiesmelfie · 2024-08-01T19:04:07Z

roles/telemetry_autoscaling/tasks/configure_heat.yml

+        line: |
+          '<testcase classname="openstack-observability"  name="configure-hest-template-automatically-scaling_instances"  time="1"/>'
+          '<failure> configure Heat template failed <failure/>'</testcase>'
+        insertbefore: "</testsuite>"
+
+    - name: increment number of failures
+      set_fact:
+        number_of_failures: "{{ number_of_failures | int + 1 }}"
+
+- name: increment number of tests
+  set_fact:
+    number_of_tests: "{{ number_of_tests | int + 1 }}"
+


Can these be done in v2_playbook_on_stats?

elfiesmelfie · 2024-08-01T19:06:41Z

callback_plugins/custom_logger.py

@@ -1,92 +1,274 @@
+# (c) 2016 Matt Clay <[email protected]>
+# (c) 2017 Ansible Project
+# GNU General Public License v3.0+ (see COPYING or https://www.gnu.org/licenses/gpl-3.0.txt)


Copy/pasting the existing junitXML plugin is not the way to go.

Either try to contribute upstream to it, or inherit from the existing class here to make changes.

There MIGHT be an issue with using a GPL-licensed library to provide the base class, since this project is not GPL licensed, nor do we plan to adopt GPL v3 as the license.

Yet another alternative would be to record the results and then render the XML at the end using a jinja2 template, so that there's no back-tracking through the document. However, this would use more memory.

elfiesmelfie · 2024-08-01T19:15:30Z

roles/telemetry_autoscaling/tasks/configure_heat.yml

+    - name: Add pass to test results
+      delegate_to: localhost
+      lineinfile:
+        path: "{{ logs_dir }}/test_run_result.xml"
+        line: '<testcase classname="openstack-observability"  name="configure-hest-template-automatically-scaling-instances"  time="1"/>'
+        insertbefore: "</testsuite>"


This is the only test result file I can see.

I need to look for the one generated by the junit plugin

elfiesmelfie · 2024-08-01T19:18:51Z

callback_plugins/custom_logger.py

-        host = result._host.get_name()
-        task_name = result._task.get_name()
-        self.log_task_result(host, 'passed', task_name)
+        self._output_dir =  os.getcwd()


This is not a sensible default. I don't know where this ends up, maybe on the executer, but it's not in a location that gets uploaded

elfiesmelfie · 2024-08-01T19:20:22Z

callback_plugins/custom_logger.py

+        self._fail_on_ignore = os.getenv('JUNIT_FAIL_ON_IGNORE', 'False').lower()
+        self._include_setup_tasks_in_report = os.getenv('JUNIT_INCLUDE_SETUP_TASKS_IN_REPORT', 'False').lower()
+        self._hide_task_arguments = os.getenv('JUNIT_HIDE_TASK_ARGUMENTS', 'True').lower()
+        self._test_case_prefix = os.getenv('JUNIT_TEST_CASE_PREFIX', 'RHOSO')


I think that this needs further work. Using prefix only doesn't work if the same task has multiple test IDs in it.

elfiesmelfie

I ran this locally, using the test playbook for the custom logger

I left ci/ansible.cfg as-is, and used the following command to test cd ci && ansible-playbook -v github/test_logger.yml

The output was this:

cat *.xml
<?xml version="1.0" ?>
<testsuites disabled="0" errors="0" failures="0" tests="1" time="0.012203693389892578125">
	<testsuite disabled="0" errors="0" failures="0" name="test_logger" skipped="0" tests="1" time="0.012203693389892578125">
		<testcase classname="/home/emma/repos/github/infrawatch/feature-verification-tests/ci/github/test_logger.yml:6" name="[localhost] Run tests to make sure the custom_logger plugin is running as expected: RHOSO-01 show some message" time="0.012203693389892578">
			<system-out>{
    &quot;changed&quot;: false,
    &quot;msg&quot;: &quot;You can start the task name with the test ID.&quot;
}</system-out>
		</testcase>
	</testsuite>
</testsuites>

It looks like unexpected values are being used in unexpected places, and the TestCase class probably needs to be updated.
There's also only one testcase recorded because there's only one task with the ID at the start of the name.

elfiesmelfie · 2024-08-01T20:51:55Z

callback_plugins/custom_logger.py

+        self._task_class = os.getenv('JUNIT_TASK_CLASS', 'False').lower()
+        self._task_relative_path = os.getenv('JUNIT_TASK_RELATIVE_PATH', '')
+        self._fail_on_change = os.getenv('JUNIT_FAIL_ON_CHANGE', 'False').lower()
+        self._fail_on_ignore = os.getenv('JUNIT_FAIL_ON_IGNORE', 'False').lower()


This should be true, because we use "ignore_errors: true" in our playbooks

elfiesmelfie

We can extend the existing xml plugin, and redefine only what we need.
It should be possible to create a new class, with the existing XML plugin as a base and then redefines the methods we need to update.

softwarefactory-project-zuul · 2024-08-07T07:59:08Z

Merge Failed.

This change or one of its cross-repo dependencies was unable to be automatically merged with the current state of its repository. Please rebase the change and upload a new patchset.
Warning:
Error merging github.com/infrawatch/feature-verification-tests for 132,3e3e2b8e43c5cb72b5c0cb9f0f46b8f7d69a57cf

elfiesmelfie · 2024-08-08T16:43:35Z

callback_plugins/custom_junit.py

@@ -0,0 +1,49 @@
+from ansible.plugins.callback import CallbackModule as OriginalCallbackModule


Use ansible.plugins.callback.junit.CallbackModule as the superclass

elfiesmelfie · 2024-08-08T16:44:18Z

callback_plugins/custom_junit.py

+from ansible.module_utils._text import to_bytes, to_text
+from ansible.utils._junit_xml import TestCase, TestError, TestFailure, TestSuite, TestSuites
+
+class CustomCallbackModule(OriginalCallbackModule):


CallbackModule is a required classname for ansible to run the module.

softwarefactory-project-zuul · 2024-08-09T15:39:38Z

Merge Failed.

This change or one of its cross-repo dependencies was unable to be automatically merged with the current state of its repository. Please rebase the change and upload a new patchset.
Warning:
Error merging github.com/infrawatch/feature-verification-tests for 132,e7b5b1eb56694d4cdda3d4ce42b9a3bd0aa93c45

softwarefactory-project-zuul · 2024-08-09T18:11:59Z

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://softwarefactory-project.io/zuul/t/rdoproject.org/buildset/0c7dd74db81a43028204e648edaadcb0

✔️ feature-verification-tests-noop SUCCESS in 4s
✔️ openstack-k8s-operators-content-provider SUCCESS in 1h 18m 46s
❌ functional-tests-on-osp18 FAILURE in 1h 04m 17s

softwarefactory-project-zuul · 2024-08-13T15:28:15Z

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://softwarefactory-project.io/zuul/t/rdoproject.org/buildset/15f121dab19441cda6f4bcc84d365aa5

✔️ feature-verification-tests-noop SUCCESS in 5s
✔️ openstack-k8s-operators-content-provider SUCCESS in 2h 08m 51s
❌ functional-tests-on-osp18 FAILURE in 1h 03m 44s

softwarefactory-project-zuul · 2024-08-13T17:31:40Z

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://softwarefactory-project.io/zuul/t/rdoproject.org/buildset/abdfd613a7144e99a17b7380cb3eb14b

✔️ feature-verification-tests-noop SUCCESS in 4s
✔️ openstack-k8s-operators-content-provider SUCCESS in 1h 22m 11s
❌ functional-tests-on-osp18 FAILURE in 1h 00m 55s

softwarefactory-project-zuul · 2024-08-14T13:29:34Z

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://softwarefactory-project.io/zuul/t/rdoproject.org/buildset/47ac0c5c769f457f8f3111222c249bef

✔️ feature-verification-tests-noop SUCCESS in 4s
❌ openstack-k8s-operators-content-provider FAILURE in 13m 11s
⚠️ functional-tests-on-osp18 SKIPPED Skipped due to failed job openstack-k8s-operators-content-provider

elfiesmelfie · 2024-08-14T16:08:51Z

recheck

softwarefactory-project-zuul · 2024-08-14T17:56:42Z

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://softwarefactory-project.io/zuul/t/rdoproject.org/buildset/97af9659e8c7442f933f22d5df65f300

✔️ feature-verification-tests-noop SUCCESS in 4s
✔️ openstack-k8s-operators-content-provider SUCCESS in 1h 31m 57s
❌ functional-tests-on-osp18 FAILURE in 1h 16m 34s

softwarefactory-project-zuul · 2024-08-16T19:12:26Z

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://softwarefactory-project.io/zuul/t/rdoproject.org/buildset/d9453489e33749a5ab3a54abc929a062

✔️ feature-verification-tests-noop SUCCESS in 4s
✔️ openstack-k8s-operators-content-provider SUCCESS in 1h 30m 53s
❌ functional-tests-on-osp18 FAILURE in 1h 16m 28s

The original test ID that custom_logger uses is still in the name, Custom logger will record test results as long as the expected test ids are in the name somewhere.

Use copy with remote src instead of shell/cp, so that the files can be copied from compute nodes into the controller logs dir Add a new play to make sure the output dir is created on the controller before copying the files.

Rename the files using the hostname of the src

softwarefactory-project-zuul · 2024-10-01T16:59:02Z

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://softwarefactory-project.io/zuul/t/rdoproject.org/buildset/9f2fafd5337146e0b77da991fd146755

✔️ feature-verification-tests-noop SUCCESS in 5s
✔️ openstack-k8s-operators-content-provider SUCCESS in 2h 20m 22s
❌ functional-tests-on-osp18 FAILURE in 1h 37m 40s
✔️ functional-logging-tests-osp18 SUCCESS in 1h 04m 20s

softwarefactory-project-zuul · 2024-10-02T12:06:29Z

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://softwarefactory-project.io/zuul/t/rdoproject.org/buildset/78ad75d4b36c420a960550cdda1e0c09

✔️ feature-verification-tests-noop SUCCESS in 4s
✔️ openstack-k8s-operators-content-provider SUCCESS in 1h 45m 49s
❌ functional-tests-on-osp18 FAILURE in 1h 32m 53s
✔️ functional-logging-tests-osp18 SUCCESS in 1h 00m 28s

elfiesmelfie reviewed Jul 23, 2024

View reviewed changes

elfiesmelfie marked this pull request as ready for review July 23, 2024 15:14

elfiesmelfie marked this pull request as draft July 23, 2024 15:14

elfiesmelfie reviewed Aug 1, 2024

View reviewed changes

elfiesmelfie reviewed Aug 6, 2024

View reviewed changes

elfiesmelfie mentioned this pull request Aug 8, 2024

[WIP] Add jUNitXML plugin to create the output xml file #136

Closed

elfiesmelfie reviewed Aug 8, 2024

View reviewed changes

elfiesmelfie mentioned this pull request Aug 15, 2024

[telemetry_autoscaling] Verify that the required custom resources are ready #139

Merged

elfiesmelfie and others added 26 commits October 1, 2024 15:02

Update hosts for report_result add some notes

57ee3b5

[ci] Remove unneeded tasks from run_autoscaling_osp18

cea60e4

[custom_junit] Update prefix for tests

24b0d36

[telemetry_autoscaling] Update test names to use prefix for junit_xml

7f6ae2d

The original test ID that custom_logger uses is still in the name, Custom logger will record test results as long as the expected test ids are in the name somewhere.

Update debugging

eec8ff9

Update report_result

02719d6

Use copy with remote src instead of shell/cp, so that the files can be copied from compute nodes into the controller logs dir Add a new play to make sure the output dir is created on the controller before copying the files.

[custom_junit] Add debug info (fix typo)

01d58c5

Fix syntax errors in custom_junit

fa11cd1

update src filename

b258f37

[report_result] Update the copied files

7c9d7c9

Rename the files using the hostname of the src

report_result inventory_host -> inventory_hostname

ab847dd

Update github/test_logger.yml

497ad3b

update custom_junit

df49b92

Update when the task name is mutated

fbb6205

Update test_logger.yml playbook

ce63797

[custom_junit] Try a zero-length prefix to record all the tasks

1483e7a

Get more info from custom_junit

a84d458

Remove extra dir

bbbff5d

Fix Exception

a280fd3

Update report_result

cda817f

Fix typo

44ab689

custom_junit: Use a blank prefix so all the tasks get recorded

b3956e6

Remove prefix from tasks

3f38aad

Comment out some prints use ignore_errors with autoscaling tests

01506d0

[custom_junit] Record the outputs for prefix IN task_name

8aef4fb

Add prefix to test tasks

3f163bf

elfiesmelfie force-pushed the mgirgis-OSPRH-8378 branch from ea2e1f1 to 3f163bf Compare October 1, 2024 14:10

[telemetry_logging] Add TEST prefix to tasks

d060198

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add a custom_junit callback plugin #132

Add a custom_junit callback plugin #132

mgirgisf commented Jul 22, 2024 •

edited by elfiesmelfie

Loading

softwarefactory-project-zuul bot commented Jul 22, 2024

mgirgisf commented Jul 23, 2024

softwarefactory-project-zuul bot commented Jul 23, 2024

elfiesmelfie Jul 23, 2024 •

edited

Loading

softwarefactory-project-zuul bot commented Jul 24, 2024

softwarefactory-project-zuul bot commented Jul 25, 2024

softwarefactory-project-zuul bot commented Jul 25, 2024

softwarefactory-project-zuul bot commented Aug 1, 2024

elfiesmelfie Aug 1, 2024

elfiesmelfie Aug 1, 2024

elfiesmelfie Aug 1, 2024

elfiesmelfie Aug 1, 2024

elfiesmelfie Aug 1, 2024

elfiesmelfie Aug 1, 2024

elfiesmelfie left a comment •

edited

Loading

elfiesmelfie Aug 1, 2024

elfiesmelfie left a comment

softwarefactory-project-zuul bot commented Aug 7, 2024

elfiesmelfie Aug 8, 2024

elfiesmelfie Aug 8, 2024

softwarefactory-project-zuul bot commented Aug 9, 2024

softwarefactory-project-zuul bot commented Aug 9, 2024

softwarefactory-project-zuul bot commented Aug 13, 2024

softwarefactory-project-zuul bot commented Aug 13, 2024

softwarefactory-project-zuul bot commented Aug 14, 2024

elfiesmelfie commented Aug 14, 2024

softwarefactory-project-zuul bot commented Aug 14, 2024

softwarefactory-project-zuul bot commented Aug 16, 2024

softwarefactory-project-zuul bot commented Oct 1, 2024

softwarefactory-project-zuul bot commented Oct 2, 2024

		@@ -0,0 +1,49 @@
		from ansible.plugins.callback import CallbackModule as OriginalCallbackModule

Add a custom_junit callback plugin #132

Are you sure you want to change the base?

Add a custom_junit callback plugin #132

Conversation

mgirgisf commented Jul 22, 2024 • edited by elfiesmelfie Loading

softwarefactory-project-zuul bot commented Jul 22, 2024

mgirgisf commented Jul 23, 2024

softwarefactory-project-zuul bot commented Jul 23, 2024

elfiesmelfie Jul 23, 2024 • edited Loading

Choose a reason for hiding this comment

softwarefactory-project-zuul bot commented Jul 24, 2024

softwarefactory-project-zuul bot commented Jul 25, 2024

softwarefactory-project-zuul bot commented Jul 25, 2024

softwarefactory-project-zuul bot commented Aug 1, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

elfiesmelfie left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

elfiesmelfie left a comment

Choose a reason for hiding this comment

softwarefactory-project-zuul bot commented Aug 7, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

softwarefactory-project-zuul bot commented Aug 9, 2024

softwarefactory-project-zuul bot commented Aug 9, 2024

softwarefactory-project-zuul bot commented Aug 13, 2024

softwarefactory-project-zuul bot commented Aug 13, 2024

softwarefactory-project-zuul bot commented Aug 14, 2024

elfiesmelfie commented Aug 14, 2024

softwarefactory-project-zuul bot commented Aug 14, 2024

softwarefactory-project-zuul bot commented Aug 16, 2024

softwarefactory-project-zuul bot commented Oct 1, 2024

softwarefactory-project-zuul bot commented Oct 2, 2024

mgirgisf commented Jul 22, 2024 •

edited by elfiesmelfie

Loading

elfiesmelfie Jul 23, 2024 •

edited

Loading

elfiesmelfie left a comment •

edited

Loading