vSphere IPI SCOS templates on 4.17

[nate-duke]

Hi All,

We updated our testing cluster from 4.14 all the way to 4.17 over the holidays (worked great!). Now that we're on the current release we're doing some operational testing around our various MachineSets and it seems that the vSphere template used to create new nodes is still FCOS and isn't able to produce viable nodes in 4.17.

I'm happy to provide any and all diagnostics needed to look into this.

To be clear the template i'm referrring to is in the okd docs
https://okd.io/docs/project/guides/vsphere-ipi#miscellaneous

disabled virtual machine with name {cluster name}-rhcos-{generated cluster id} which using as template for further scaling

[jcpowermac]

@nate-duke since you upgraded the workaround that was added to the installer isn't there. Disable secure boot on the template and retry. Templates are never upgraded by OCP but there is future work that it could be possible.

[nate-duke]

Secure boot is disabled across the board. We wouldn't have made it through the update w/o that.

N.B: these are BIOS booting, not EFI.

[nate-duke]

as an update from some discussion in slack it seems what's failing is the machine-config-daemon-firstboot.service:

Jan 02 13:48:01 localhost podman[1283]: I0102 13:48:01.062681    1327 daemon.go:521] assuming we can use container binary chroot() to host
Jan 02 13:48:01 localhost podman[1283]: I0102 13:48:01.169582    1327 daemon.go:571] Invoking re-exec /run/bin/machine-config-daemon
Jan 02 13:48:01 localhost podman[1283]: /run/bin/machine-config-daemon: /lib64/libc.so.6: version `GLIBC_2.34' not found (required by /run/bin/machine-config-daemon)
Jan 02 13:48:01 localhost podman[1283]: 2025-01-02 13:48:01.204159769 +0000 UTC m=+0.360938186 container died 5d24ff69e0aeac7980ec4c2ff286a9a48666de2d47bd10387cf628b8732>
Jan 02 13:48:01 localhost podman[1283]: 2025-01-02 13:48:01.249628315 +0000 UTC m=+0.406406749 container remove 5d24ff69e0aeac7980ec4c2ff286a9a48666de2d47bd10387cf628b87>
Jan 02 13:48:01 localhost systemd[1]: machine-config-daemon-firstboot.service: Main process exited, code=exited, status=1/FAILURE
Jan 02 13:48:01 localhost systemd[1]: machine-config-daemon-firstboot.service: Failed with result 'exit-code'.
Jan 02 13:48:01 localhost systemd[1]: Failed to start Machine Config Daemon Firstboot.

Other details from the booted FCOS VM:

[root@localhost ~]# cat /etc/os-release;rpm-ostree status; /lib64/libc.so.6
NAME=Fedora
VERSION="34.20210626.3.1 (CoreOS)"
ID=fedora
VERSION_ID=34
VERSION_CODENAME=""
PLATFORM_ID="platform:f34"
PRETTY_NAME="Fedora CoreOS 34.20210626.3.1"
ANSI_COLOR="0;38;2;60;110;180"
LOGO=fedora-logo-icon
CPE_NAME="cpe:/o:fedoraproject:fedora:34"
HOME_URL="https://getfedora.org/coreos/"
DOCUMENTATION_URL="https://docs.fedoraproject.org/en-US/fedora-coreos/"
SUPPORT_URL="https://github.com/coreos/fedora-coreos-tracker/"
BUG_REPORT_URL="https://github.com/coreos/fedora-coreos-tracker/"
REDHAT_BUGZILLA_PRODUCT="Fedora"
REDHAT_BUGZILLA_PRODUCT_VERSION=34
REDHAT_SUPPORT_PRODUCT="Fedora"
REDHAT_SUPPORT_PRODUCT_VERSION=34
PRIVACY_POLICY_URL="https://fedoraproject.org/wiki/Legal:PrivacyPolicy"
VARIANT="CoreOS"
VARIANT_ID=coreos
OSTREE_VERSION='34.20210626.3.1'
DEFAULT_HOSTNAME=localhost
State: idle
Deployments:
● ostree://fedora:fedora/x86_64/coreos/stable
                   Version: 34.20210626.3.1 (2021-07-14T14:49:01Z)
                    Commit: 252fffde6f56d183a3c51c05a0c602b61011f6cb4de23a58313ba3b0023dc360
              GPGSignature: Valid signature by 8C5BA6990BDB26E19F2A1A801161AE6945719A39
GNU C Library (GNU libc) release release version 2.33.
Copyright (C) 2021 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.
There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A
PARTICULAR PURPOSE.
Compiled by GNU CC version 11.1.1 20210531 (Red Hat 11.1.1-3).
libc ABIs: UNIQUE IFUNC ABSOLUTE
For bug reporting instructions, please see:
<https://www.gnu.org/software/libc/bugs.html>.

[nate-duke]

Ok this is sorted out. For future searchers what we needed to do was:

0. record name of existing template, will be something like CLUSTERNAME-rhcos-CLUSTERID. in my case it was dev-rhcos-njkpp
1. get new FCOS distribution OVA (see below)
2. rename existing rhcos (actually FCOS) template to keep for posterity
3. upload OVA to vcenter
4. Edit created VM to disable secure boot (or just switch from EFI to BIOS)
5. Clone VM to template named the same as what you recorded in step 0
6. Test

For Reference:
FCOS OVA artifacts: in openshift/installer/data/data/coreos/fcos.json

N.B.
This likely will not affect NEWISHly installed clusters. Ours was particularly old and had a creaky version of glibc that newer centos containers didn't like too much.