Upload container logs for live tests

[AlanCoding]

SUMMARY

Looking at #15705, it seems the task manager isn't running, ping @jessicamack

We need to see the logs to get hints as to what is happening, but the tests don't upload the logs.

The "live" tests are weird, sometimes server logs are emitted in the pytest process, which are captured in the test output. But it's split, because some work is ping-ponged to the task system, so those logs can only be obtained by asynchronous getting the server logs after the tests finish.

This is already a solved problem, but wasn't solved for these tests.

ISSUE TYPE

• Bug, Docs Fix or other nominal change

COMPONENT NAME

• API

[AlanCoding]

The output is inside the "docker-compose-logs" artifact, which extracts as a folder.

Current name is live-tests-.log. Has some stuff like:

2024-12-18 19:42:44,421 WARN exited: awx-rsyslogd (exit status 1; not expected)
2024-12-18 19:42:44,422 INFO spawned: 'awx-rsyslogd' with pid 2114
rsyslogd: could not open config file '/var/lib/awx/rsyslog/rsyslog.conf': Permission denied [v8.2102.0-106.el9 try https://www.rsyslog.com/e/2104 ]
rsyslogd: run failed with error -2104 (see rsyslog.h or try https://www.rsyslog.com/e/2104 to learn what that number means)
2024-12-18 19:42:44,430 INFO success: awx-rsyslogd entered RUNNING state, process has stayed up for > than 0 seconds (startsecs)
2024-12-18 19:42:44,430 WARN exited: awx-rsyslogd (exit status 1; not expected)
awx-rsyslogd stderr | rsyslogd: could not open config file '/var/lib/awx/rsyslog/rsyslog.conf': Permission denied [v8.2102.0-106.el9 try https://www.rsyslog.com/e/2104 ]
awx-rsyslogd stderr | rsyslogd: run failed with error -2104 (see rsyslog.h or try https://www.rsyslog.com/e/2104 to learn what that number means)
awx-rsyslogd stderr | 

[AlanCoding]

I got rid of the trailing - character because the stuff after that failed to template anything (was blank).

Once the checks pass here I'll be happy to merge this.

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

[jessicamack]

The output is inside the "docker-compose-logs" artifact, which extracts as a folder.

Current name is live-tests-.log. Has some stuff like:

2024-12-18 19:42:44,421 WARN exited: awx-rsyslogd (exit status 1; not expected)
2024-12-18 19:42:44,422 INFO spawned: 'awx-rsyslogd' with pid 2114
rsyslogd: could not open config file '/var/lib/awx/rsyslog/rsyslog.conf': Permission denied [v8.2102.0-106.el9 try https://www.rsyslog.com/e/2104 ]
rsyslogd: run failed with error -2104 (see rsyslog.h or try https://www.rsyslog.com/e/2104 to learn what that number means)
2024-12-18 19:42:44,430 INFO success: awx-rsyslogd entered RUNNING state, process has stayed up for > than 0 seconds (startsecs)
2024-12-18 19:42:44,430 WARN exited: awx-rsyslogd (exit status 1; not expected)
awx-rsyslogd stderr | rsyslogd: could not open config file '/var/lib/awx/rsyslog/rsyslog.conf': Permission denied [v8.2102.0-106.el9 try https://www.rsyslog.com/e/2104 ]
awx-rsyslogd stderr | rsyslogd: run failed with error -2104 (see rsyslog.h or try https://www.rsyslog.com/e/2104 to learn what that number means)
awx-rsyslogd stderr | 

what could have caused the permission issue here? it was fine previously and I don't remember any work around that being merged recently?

[AlanCoding]

I don't know. Pasting some from output:

2024-12-18T20:03:52.3629437Z #49 [stage-2 19/34] ADD tools/ansible/roles/dockerfile/files/rsyslog.conf /var/lib/awx/rsyslog/rsyslog.conf
2024-12-18T20:03:52.3630015Z #49 CACHED

and later

2024-12-18T20:03:52.3595186Z #23 [stage-2 33/34] RUN for dir in       /var/lib/awx       /var/lib/awx/rsyslog       /var/lib/awx/rsyslog/conf.d       /var/lib/awx/.local/share/containers/storage       /var/run/awx-rsyslog       /var/log/nginx       /var/lib/pgsql       /var/run/supervisor       /var/run/awx-receptor       /var/lib/nginx ;     do mkdir -m 0775 -p $dir ; chmod g+rwx $dir ; chgrp root $dir ; done &&     for file in       /etc/subuid       /etc/subgid       /etc/group       /etc/passwd       /var/lib/awx/rsyslog/rsyslog.conf ;     do touch $file ; chmod g+rw $file ; chgrp root $file ; done
2024-12-18T20:03:52.3596871Z #23 CACHED
2024-12-18T20:03:52.3596965Z 

Locally I can confirm

bash-5.1$ ls -la /var/lib/awx/rsyslog/rsyslog.conf
-rw-r-----. 1 awx root 351 Dec 18 20:44 /var/lib/awx/rsyslog/rsyslog.conf

That all seems okay, I can read this as awx user.