[Build 202502241510-4.3] openQA test fails in startup - irq 23: nobody cared

[marmarek]

Observation

openQA test in scenario qubesos-4.3-kernel-x86_64-system_tests_network@64bit fails in
startup

[   43.313712] irq 23: nobody cared (try booting with the "irqpoll" option)
[   43.315087] CPU: 1 UID: 0 PID: 0 Comm: swapper/1 Not tainted 6.14.0-0.rc4.1.qubes.1.fc41.x86_64 #1
[   43.315090] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS rel-1.16.3-2-gc13ff2cd-prebuilt.qemu.org 04/01/2014
[   43.315091] Call Trace:
[   43.315092]  <IRQ>
[   43.315094]  dump_stack_lvl+0x5d/0x80
[   43.315104]  __report_bad_irq+0x35/0xa7
[   43.315106]  note_interrupt.cold+0xa/0x67
[   43.315107]  handle_irq_event+0x6f/0x90
[   43.315110]  handle_fasteoi_irq+0x78/0x200
[   43.315112]  generic_handle_irq+0x3f/0x60
[   43.315113]  __evtchn_fifo_handle_events+0x1e0/0x2c0
[   43.315117]  xen_evtchn_do_upcall+0x6e/0xc0
[   43.315119]  __xen_pv_evtchn_do_upcall+0x21/0x30
[   43.315121]  xen_pv_evtchn_do_upcall+0x84/0xa0
[   43.315123]  </IRQ>
[   43.315123]  <TASK>
[   43.315124]  exc_xen_hypervisor_callback+0x8/0x20
[   43.315126] RIP: e030:xen_hypercall_pv+0x9/0x20
[   43.315128] Code: 00 00 00 c3 cc cc cc cc 66 0f 1f 84 00 00 00 00 00 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 f3 0f 1e fa 51 41 53 0f 05 <41> 5b 59 c3 cc cc cc cc 66 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 40
[   43.315129] RSP: e02b:ffffc900400c3ec0 EFLAGS: 00000246
[   43.315131] RAX: 0000000000000000 RBX: 0000000000000000 RCX: ffffffff815e6099
[   43.315132] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000001
[   43.315133] RBP: ffff88810086d200 R08: 00000000000fac3c R09: 0000000000000000
[   43.315133] R10: 0000000000007ff0 R11: 0000000000000246 R12: 0000000000000000
[   43.315134] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000
[   43.315135]  ? xen_hypercall_pv+0x9/0x20
[   43.315137]  xen_safe_halt+0x15/0x20
[   43.315138]  default_idle+0x9/0x20
[   43.315139]  default_idle_call+0x29/0x100
[   43.315140]  cpuidle_idle_call+0x123/0x160
[   43.315144]  do_idle+0x78/0xd0
[   43.315145]  cpu_startup_entry+0x29/0x30
[   43.315146]  cpu_bringup_and_idle+0x18/0x20
[   43.315148]  asm_cpu_bringup_and_idle+0x9/0x10
[   43.315152]  </TASK>
[   43.315152] handlers:
[   43.342272] [<00000000bcb67e39>] xen_pcibk_guest_interrupt [xen_pciback]
[   43.343272] Disabling IRQ #23

Test suite description

Reproducible

Fails since (at least) Build 202502241510-4.3 (current job)

Expected result

Last good: 202502211025-4.3 (or more recent)

Further details

Always latest result in this scenario: latest

This started happening some time ago. While the message itself doesn't look to be fatal, it looks to correlate with a system crash soon after (without any panic message on the serial console - system just resets itself). This happens only when running inside KVM.

[marmarek]

Affected jobs, and a worker on which it happened:

job url	worker host
https://openqa.qubes-os.org/tests/00126263	"ramen"
https://openqa.qubes-os.org/tests/00126313	"ramen"
https://openqa.qubes-os.org/tests/00126378	"ramen"
https://openqa.qubes-os.org/tests/00126461	"ramen"
https://openqa.qubes-os.org/tests/00126726	"ramen"
https://openqa.qubes-os.org/tests/00126754	"ramen"
https://openqa.qubes-os.org/tests/00126869	"ramen"
https://openqa.qubes-os.org/tests/00126911	"ramen"
https://openqa.qubes-os.org/tests/00126915	"ramen"
https://openqa.qubes-os.org/tests/00126946	"ramen"
https://openqa.qubes-os.org/tests/00127146	"ramen"
https://openqa.qubes-os.org/tests/00127232	"ramen"
https://openqa.qubes-os.org/tests/00127694	"ramen"
https://openqa.qubes-os.org/tests/00127701	"ramen"
https://openqa.qubes-os.org/tests/00127964	"ramen"
https://openqa.qubes-os.org/tests/00128101	"ramen"
https://openqa.qubes-os.org/tests/00128124	"ramen"
https://openqa.qubes-os.org/tests/00128217	"ramen"
https://openqa.qubes-os.org/tests/00128230	"ramen"
https://openqa.qubes-os.org/tests/00128236	"ramen"
https://openqa.qubes-os.org/tests/00128261	"ramen"
https://openqa.qubes-os.org/tests/00128274	"ramen"
https://openqa.qubes-os.org/tests/00128279	"ramen"
https://openqa.qubes-os.org/tests/00128508	"ramen"
https://openqa.qubes-os.org/tests/00128636	"ramen"
https://openqa.qubes-os.org/tests/00128788	"ramen"
https://openqa.qubes-os.org/tests/00128865	"ramen"
https://openqa.qubes-os.org/tests/00128902	"ramen"
https://openqa.qubes-os.org/tests/00129005	"ramen"
https://openqa.qubes-os.org/tests/00129083	"ramen"
https://openqa.qubes-os.org/tests/00129244	"ramen"
https://openqa.qubes-os.org/tests/00129780	"ramen"
https://openqa.qubes-os.org/tests/00129863	"ramen"

So, looks to be specific to a single worker.

[marmarek]

Interestingly, "onigiri" is identical hardware, same software, same firmware version etc, and the issue doesn't appear there. Could be a faulty hardware.