Adding support for performing updates from a container image

[rdoxenham]

In this PR I'm introducing support for transactional-update to be able to consume a container image (or OCI artefact) as the source for the next boot snapshot. This is very important functionality that allows customers to build, distribute, and validate their operating system images via a standardised container image workflow; they can build OS images via Dockerfiles, store and retrieve them via a standard image registry, and put them through standard SBOM and vulnerability checkers.

This PR is specifically addressing requirements to enable customers to upgrade an existing machine, regardless of how it was deployed, to a new image-based snapshot, however the initial day1 approach could use tooling such as kiwi-ng system stackbuild to build bootable raw and SelfInstall ISO's based on the same container image, which is already a supported feature.

The approach taken in this PR:

• Enables an easy on-ramp (and off-ramp) for users wanting to use the image approach
• Leverages existing standard tooling to achieve the workflow; this would just use the standard OS layout/tools.
• Enables users to switch between standard package or image based upgrades (no need to build a new image every time)
• Doesn't break existing workflow for patching or migrating between products
• Retains full snapper (i.e. transactional-update rollback) functionality
• Retains full customisation options through ignition/combustion and jeos-firstboot (provided they're in the image!)
• Preserves existing user configuration between updates/rollbacks [1]

I could use some help with ensuring that I'm making the correct calls, as I'm mixing tukit callext with using the {SNAPSHOT_DIR} directly. I suspect there's a more elegant/safer/better way of achieving this, hence why it's labelled as a work in progress for now.

Further, we likely need to run some validations to make sure that the target image is actually bootable. This code forces a dracut and grub2-mkconfig run, but I can see this being an area where it may be trivial to make the system difficult to boot and rescue. I'm not an expert here, so I'm wondering whether there are checks that we could execute, and if they fail, we can abort the snapshot. Documentation on how to build an image will be very important, especially as it relates to partitions (or btrfs subvolumes that are ignored) but we should likely do some additional sanity checks to verify the state of the new snapshot rather than blindly closing it and enabling the user to reboot, where we're not able to provide a decent level of confidence in a successful reboot.

For testing, I used this on-top of SLE Micro 5.5 and used a test container image available at (registry.opensuse.org/home/roxenham/slemicro/containers/edge/sle-micro/5.5:20240726). This is a very simple image that aims to mirror the standard SLE Micro 5.5 packages as defined in the original Kiwi image definition file, i.e. it's not cut down, but has the bare set of packages typically installed, or as defined in the "Default" profile. Of course, it's perfectly possible to modify this image to suit, e.g. adding a package to it is as simple as building and pushing a new image (noting that the suseconnect is only required for commercially registered images, this wouldn't be required for Leap Micro or MicroOS:

% cat Dockerfile
FROM registry.opensuse.org/home/roxenham/slemicro/containers/edge/sle-micro/5.5:20240726
RUN suseconnect -r <regcode> && zypper --gpg-auto-import-keys ref \
     && zypper in -y nvidia-open-driver-G06-signed-kmp-default \
     && suseconnect -d && zypper clean -a

ARG IMAGE_REPO=unknown
ARG IMAGE=unknown
ARG IMAGE_TAG=unknown

RUN sed -i '/IMAGE/d' /usr/lib/os-release && \
    sed -i '/TIMESTAMP/d' /usr/lib/os-release && \
    echo IMAGE_REPO=\"$IMAGE_REPO\"              >> /usr/lib/os-release && \
    echo IMAGE_TAG=\"$IMAGE_TAG\"                >> /usr/lib/os-release && \
    echo IMAGE=\"$IMAGE_REPO:$IMAGE_TAG\"        >> /usr/lib/os-release && \
    echo TIMESTAMP="`date +'%Y%m%d%H%M%S'`"      >> /usr/lib/os-release

% podman build -t harbor.rancher.rdoxenham.com/slemicro/5.5:$(date +'%Y%m%d') \
    --build-arg "IMAGE=harbor.rancher.rdoxenham.com/slemicro/5.5:$(date +'%Y%m%d')" \
    --build-arg "IMAGE_TAG=$(date +'%Y%m%d')" \
    --build-arg "IMAGE_REPO=harbor.rancher.rdoxenham.com/slemicro/5.5" .

% podman push harbor.rancher.rdoxenham.com/slemicro/5.5:20240727
(...)

Then, this image can be used as an input to code provided as part of this PR:

# transactional-update apply-oci --image harbor.rancher.rdoxenham.com/slemicro/5.5:20240727
(...)

[after reboot]

# cat /etc/os-release
NAME="SLE Micro"
VERSION="5.5"
VERSION_ID="5.5"
PRETTY_NAME="SUSE Linux Enterprise Micro 5.5"
ID="sle-micro"
ID_LIKE="suse"
ANSI_COLOR="0;32"
CPE_NAME="cpe:/o:suse:sle-micro:5.5"
GRUB_ENTRY_NAME="SLE Micro"
IMAGE_REPO="harbor.rancher.rdoxenham.com/slemicro/5.5"
IMAGE_TAG="20240727"
IMAGE="harbor.rancher.rdoxenham.com/slemicro/5.5:20240727"
TIMESTAMP=20240727173117

# rpm -qa | grep nvidia-open-driver
nvidia-open-driver-G06-signed-kmp-default-550.90.07_k5.14.21_150500.55.65-150500.3.47.1.x86_64

Some further guidance from the TU community would be appreciated. Thanks a lot!

[1] This approach aligns well with the bootc project (https://containers.github.io/bootc/filesystem.html#etc) to be able to persist /etc configuration across image updates, which would enable configuration such as OS registration, package repositories, static network configuration, and various other components to persist, and not be overwritten by the OS upgrade. However, one question we may want to answer is whether we want to enable /usr/etc to enable specific files in /etc to be overwritten by force by contents found in /usr/etc to give users a release-valve for changing certain persistent configuration over time as part of a 3-way merge (this PR doesn't do this yet).

[lz-coder]

any plans for this to be merged?

[laenion]

any plans for this to be merged?

Yes, I just have to find the time to finally review it! Sorry for the delay!

[joostwestra]

registry.opensuse.org/home/roxenham/slemicro/containers/edge/sle-micro/5.5:20240726 does not exist anymore?

[rdoxenham]

Hi @joostwestra - yes, that system rebuilds every day and discards already created tags, so you can either select the latest tag, or just build off latest:

• registry.opensuse.org/home/roxenham/slemicro/containers/edge/sle-micro/5.5:20240912
• registry.opensuse.org/home/roxenham/slemicro/containers/edge/sle-micro/5.5:latest

[agracey]

Any update on timeline for a review of this?

[puneetlws]

Since slemicro 6 is also released now, Is it possible to add migration functionality to it through above mechanism, from 5.5 to 6?? Right now to migrate to 6 there is 'transactional-update migrate' command which internally calls zypper migrate, which requires connectivity to SUSEConnect.

[joostwestra]

We tested this feature by manually patching it in. We think it is a valuable feature for a wide audience.
Anything we can do to help get this feature to be picked up further?

[laenion]

I'll include it in the next major release (hopefully to be released soon), it's just that I'm currently prioritizing finalizing the work on moving the /etc overlays to btrfs subvolumes.

[agracey]

@laenion Thank you!

[laenion]

Thanks a lot for the implementation! I really love the approach, though I'm not sure how bullet-proof it has to become to merge it. So let me just add a few comments.

It seems that one so far has to specify a specific image, i.e. it won't detect whether there's an update. On the other hand it's also possible to install any image (even other distributions) which will obviously break.
I guess it would make sense to either predefine (e.g. in tukit.conf or transactional-update.conf) the repository and just let the user specify the specific version by default.

In the longer term I think we also want to integrate the functionality into tukit directly as an alternative implementation to snapper. Then the updates could also be triggered by D-Bus and the API.

[laenion]

On the one hand it's good that the directories are excluded explicitly because the directories will be overmounted anyway, on the other hand transactional-update is warning at the end of the update about those files, and the created files would still be available as a reference.

Both ways are valid approaches though, I just wanted to mention this.

[laenion]

I'm not sure how this rsync call is different from the one above syncing the root file system. Isn't the result exactly the same when removing /etc from the exclude list?

I guess this rsync call is supposed to synchronize the changed files in /etc from the currently running system into the new snapshot?

[laenion]

I don't think a dedicated --help option for this specific command is useful because it would be the only command with this behavior - I'd include the information about the required parameters in the general help.

[laenion]

Instead of hardcoding this list I'd suggest to dynamically parse it using findmnt, but also see the comment where the exclude list is processed below.

[laenion]

If you wouldn't consume all arguments it would even be possible to install additional packages in one go (e.g. transactional-update apply-oci --image <path> pkg in <package>.

[laenion]

I could use some help with ensuring that I'm making the correct calls, as I'm mixing tukit callext with using the {SNAPSHOT_DIR} directly. I suspect there's a more elegant/safer/better way of achieving this, hence why it's labelled as a work in progress for now.

These calls are perfectly fine. You could use tukit call podman image pull ... and also call rsync with tukit call for syncing the initial root file system, but the result will (with the current exclusion list) be the same.

Further, we likely need to run some validations to make sure that the target image is actually bootable. This code forces a dracut and grub2-mkconfig run, but I can see this being an area where it may be trivial to make the system difficult to boot and rescue. I'm not an expert here, so I'm wondering whether there are checks that we could execute, and if they fail, we can abort the snapshot.

Indeed, also see my comment in #128 (review).

Documentation on how to build an image will be very important, especially as it relates to partitions (or btrfs subvolumes that are ignored) but we should likely do some additional sanity checks to verify the state of the new snapshot rather than blindly closing it and enabling the user to reboot, where we're not able to provide a decent level of confidence in a successful reboot.

👍