Use alma8 images on linux, while staying on 2.17 sysroot

[h-vetinari]

Since it's taken us so long to upgrade from centOS 6 to 7, we'll very quickly find ourselves in a situation that we struggled with already some time ago (see discussion in this issue): a growing number of feedstocks will require a newer glibc, and our images should provide a new enough baseline version, so that the only thing that feedstocks need to actively override is c_stdlib_version (and thus the sysroot), but nothing else.

For example, google's "foundational" support matrix defines a lower bound of glibc 2.27, meaning that things like abseil/bazel/grpc/protobuf/re2 etc. will start beginning to rely on glibc features >2.17 in the near future (and even though it's not a google project, one of the baseline dependencies of that stack is starting to require it).

We can handle the c_stdlib_version in the migrators (like we did for macOS 10.13, before the conda-forge-wide baseline was lifted), but changing more than one key of the mega-zip involving the docker images is really painful, especially if CUDA is involved (example), so having the images be alma8 by default will be very helpful there.

To a lesser extent, it will also save us from run-time bugs in older glibc versions, like we had with some broken trigonometry functions in 2.12 (before the image moved to cos7 while the sysroot still stayed at 2.12).

There are other scenarios still where this is necessary, see the discussion here for example.

While it's already possible to use alma8 images, the main thing we're blocked is the lack of CDTs for alma 8 (pending conda-forge/cdt-builds#66), c.f. conda-forge/conda-forge.github.io#1941

This issue is for tracking this effort, and any other eventual tasks that need to be resolved before doing so.

[hmaarrfk]

It would be great if we could do this for aarch + cuda12 to start. But I think we should generally move the base image forward.

xref: https://github.com/conda-forge/pytorch-cpu-feedstock/blob/main/recipe/conda_build_config.yaml#L17

[h-vetinari]

AFAIU the only remaining issue is the reduction of CDTs from cos7 to alma8; we should try to do a special "remove/replace CDTs" migration because breaking 100+ feedstocks is not really a good option, even if we provide a way to opt into the old images again.

[hmaarrfk]

Ah i see, the tk feedstock throws quite a thorn in the "rip out all the CDTs too".

[isuruf]

Using alma8 images and using an up-to-date sysroot are two different problems. In order to change to alma8 images, we just need to check to see if all the requirements in recipe/yum_requirements.txt are available in the alma8 images.

[isuruf]

It doesn't have to be an exhaustive check, but just a few feedstocks that use yum requirements to see if it really works.

[jakirkham]

Wonder if a smaller step would simply making it a bit easier for users to opt-in to using the newer images

Did a little exploration of this in PR: #6548

Likely needs more work, but maybe it is already useful for discussion/iteration at this stage