Switch from FCOS to SCOS on OKD 4.15 to 4.16 Upgrade Fails due to old e2fsck and FEATURE_C12

[boohomelabs]

Hello,

I have a six node (3 control, 3 worker) OKD 4.15.0-0.okd-2024-03-10-010116 cluster that I'm upgrading to 4.16.0-okd-scos.1 following the documentation at https://okd.io/docs/project/upgrade-notes/from-4-15/force-upgrade-to-stable-4-16.

The manual changes and interventions described therein work fine until it comes to updating the "machine-config" operator, which looks like it initiates the updates of the worker and control nodes and the whole FCOS to SCOS migration in the base OS.

From cat /etc/os-release, it looks like I'm starting from Fedora CoreOS 39.20240210.3.0 and it's trying to switch to CentOS Stream CoreOS 416.9.202411211032-0 but failing with (sanitized):

systemd[1]: Starting File System Check on /dev/disk/by-uuid/3a329d5b-3501-40c4-b535-f4b4c9df34d1...
systemd-fsck[9270]: /dev/vda3 has unsupported feature(s): FEATURE_C12
systemd-fsck[9270]: e2fsck: Get a newer version of e2fsck!
systemd-fsck[9270]: boot: ********** WARNING: Filesystem still has errors **********
systemd-fsck[9268]: fsck failed with exit status 12.
systemd[1]: systemd-fsck@dev-disk-by\x2duuid-3a329d5b\x2d3501\x2d40c4\x2db535\x2df4b4c9df34d1.service: Main process exited, >
systemd[1]: systemd-fsck@dev-disk-by\x2duuid-3a329d5b\x2d3501\x2d40c4\x2db535\x2df4b4c9df34d1.service: Failed with result 'e>
systemd[1]: Failed to start File System Check on /dev/disk/by-uuid/3a329d5b-3501-40c4-b535-f4b4c9df34d1.
systemd[1]: Dependency failed for CoreOS Dynamic Mount for /boot.
systemd[1]: Dependency failed for rpm-ostree System Management Daemon.

I believe the "FEATURE_C12" is the "orphan_file" feature that I've read in other threads. It seems as though initially the OKD nodes were partitioned/setup by FCOS and this feature was used/enabled, now that it's trying to switch to SCOS, with a lower version of tune2fs and e2fsck it doesn't know how to handle the filesystem previously made with a higher version.

There is a single line in the documentation at https://okd.io/docs/project/upgrade-notes/from-4-15/fcos-to-scos-migration that says, "We have seen reports related to ext4 features that were present on FCOS installs but not available on SCOS." so I guess this is the case here. I didn't knowingly enable FEATURE_C12 when OKD was deployed via the bootstrap method following the documentation at https://docs.okd.io/4.15/welcome/index.html for installing a Cluster on bare metal (though mine are VMs).

Does anyone know how to work around this filesystem issue to resume the upgrade process and complete the FCOS to SCOS migration (and so finish the upgrade to 4.16)? I've tried to disable the "orphan_file" feature with tune2fs but the SCOS-supplied version is too old and newer versions (e.g. from Fedora that I've scp'd across) require a newer version of libc6.

Thanks.

References/Similar Discussions and Issues:
#2050
#2041