Bug: purging fips leads to unbootable system, crashes during removal

[panlinux]

Description of the bug

Launch a multipass vm, focal. Note it booted into the generic kernel, not kvm like the lxd VMs. For some reason, and I didn't investigate why, the LXD VMs that boot into the kvm kernel by default don't exhibit this problem. But if you switch a LXD VM to use the generic kernel, and do these steps, it will also be affected.

Install fips, reboot.

tail journal logs, apt terminal in /var/log/apt/term.log

Disable fips with --purge. Notice how logs get full of crashes and errors, mostly because of libraries doing the fips selftest and failing. Even apt-get crashes.

This leads to an incomplete removal, in the sense that some steps are not done (I assume it's the package's postinst that fail), and the system won't boot, dropping you into an initramfs prompt.

I haven't fully troubleshooted this yet, but I suspect the fips disablement must be done in stages. At the very least, remove the hmac packages last, otherwise all selftests will fail like this:

Oct 23 15:51:56 f-pro-fips-multipass apt-config[22671]: Libgcrypt error: integrity check using `/lib/x86_64-linux-gnu/.libgcrypt.so.20.hmac' failed: No such file or directory
Oct 23 15:51:56 f-pro-fips-multipass apt-config[22671]: Libgcrypt notice: state transition Self-Test => Error                                                                 
(...)
Oct 23 15:52:04 f-pro-fips-multipass kernel: apt-get[22687]: segfault at 0 ip 00007fc0042b05c0 sp 00007fffb55c8c78 error 4 in libgcrypt.so.20.2.5 (deleted)[7fc004297000+d0000
]
Oct 23 15:52:04 f-pro-fips-multipass kernel: Code: 00 00 48 8b 40 48 48 85 c0 75 be 48 8d 35 c8 95 0b 00 bf 05 00 00 00 e8 9e 95 fe ff 66 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 
00 <48> 8b 07 f6 40 18 02 75 17 48 83 ec 08 e8 0e fe ff ff 31 c0 48 83

From apt's terminal log:

Removing linux-fips (5.4.0.1024.3) ...^M
Removing linux-headers-fips (5.4.0.1024.3) ...^M
Removing linux-headers-5.4.0-1024-fips (5.4.0-1024.28+recert1) ...^M
Removing linux-fips-headers-5.4.0-1024 (5.4.0-1024.28+recert1) ...^M
Removing linux-image-fips (5.4.0.1024.3) ...^M
Removing linux-modules-extra-5.4.0-1024-fips (5.4.0-1024.28+recert1) ...^M
Removing linux-image-hmac-5.4.0-1024-fips (5.4.0-1024.28+recert1) ...^M
Removing linux-image-5.4.0-1024-fips (5.4.0-1024.28+recert1) ...^M
W: Removing the running kernel^M
I: /boot/vmlinuz is now a symlink to vmlinuz-5.4.0-164-generic^M
I: /boot/initrd.img is now a symlink to initrd.img-5.4.0-164-generic^M
/etc/kernel/postrm.d/initramfs-tools:^M
update-initramfs: Deleting /boot/initrd.img-5.4.0-1024-fips^M
/etc/kernel/postrm.d/zz-update-grub:^M
../crypto/fips/fips.c:153: OpenSSL internal error: FATAL FIPS SELFTEST FAILURE^M
Aborted (core dumped)^M
Sourcing file `/etc/default/grub'^M
Sourcing file `/etc/default/grub.d/50-cloudimg-settings.cfg'^M
Sourcing file `/etc/default/grub.d/99-ubuntu-fips.cfg'^M
Sourcing file `/etc/default/grub.d/init-select.cfg'^M
Generating grub configuration file ...^M
../crypto/fips/fips.c:153: OpenSSL internal error: FATAL FIPS SELFTEST FAILURE^M
(...)
../crypto/fips/fips.c:153: OpenSSL internal error: FATAL FIPS SELFTEST FAILURE^M
libgcrypt selftest: binary  (0): No such file or directory (/lib/x86_64-linux-gnu/.libgcrypt.so.20.hmac)^M
Aborted (core dumped)^M
(...)
Removing linux-modules-5.4.0-1024-fips (5.4.0-1024.28+recert1) ...^M
Processing triggers for initramfs-tools (0.136ubuntu6.7) ...^M
update-initramfs: Generating /boot/initrd.img-5.4.0-164-generic^M
../crypto/fips/fips.c:153: OpenSSL internal error: FATAL FIPS SELFTEST FAILURE^M
Aborted (core dumped)^M
../crypto/fips/fips.c:153: OpenSSL internal error: FATAL FIPS SELFTEST FAILURE^M
Aborted (core dumped)^M
(...)

And so on.

Expected behavior

No crashes during FIPS disablement/purge, and a bootable system afterwards.

Current behavior

System is unbootable, falling into an initramfs prompt.

To Reproduce

Please include details on how to reproduce the bug.
Launch a focal multipass vm, enable fips, reboot, disable fips with purge, observe crashes in logs. If you reboot at this stage, the VM won't come back.

System information:
Pro version 30 (SRU testing)

Additional context

Right after purging fips, if you regenerate the initramfs and grub, then the system is bootable:

sudo update-initramfs -k all -c
sudo update-grub

[renanrodrigo]

Fixed in #2812 in the release branch, will be ported to main after release