[AWS] SageMaker: RAPIDS 25.02 conda packages cannot be installed on Notebook Instances (requires GLIBC 2.28+)

[jameslamb]

Description

As of this writing, Amazon SageMaker Notebook Instances only support Amazon Linux 2: https://docs.aws.amazon.com/sagemaker/latest/dg/nbi-al2.html

That distribution uses GLIBC 2.26.

As a result of rapidsai/build-planning#131, RAPIDS 25.02+ packages required GLIBC 2.28+.

Taken together, this means that RAPIDS 25.02 conda packages cannot be installed on any of the "platforms" supported by Sagemaker Notebook Instances.

Reproducible Example

I created a SageMaker Notebook Instance in us-east-2, following the docs at https://docs.rapids.ai/deployment/nightly/cloud/aws/sagemaker/.

• instance type: ml.p3.2xlarge
• "platform": Amazon Linux 2, Jupyterlab 4

Put the following into a lifecycle configuration

mamba create -y -n rapids -c rapidsai-nightly -c conda-forge -c nvidia rapids=25.02 python=3.12 cuda-version=12.5 \
    boto3 \
    ipykernel \
    'sagemaker-python-sdk>=2.239.0'

That failed, like this:

Looking for: ['rapids=25.02', 'python=3.12', 'cuda-version=12.5', 'boto3', 'ipykernel', "sagemaker-python-sdk[version='>=2.239.0']"]

conda-forge/linux-64                                        Using cache
conda-forge/noarch                                          Using cache
nvidia/linux-64                                             Using cache
nvidia/noarch                                               Using cache
pytorch/linux-64                                            Using cache
pytorch/noarch                                              Using cache
rapidsai-nightly/linux-64                                     No change
rapidsai-nightly/noarch                                       No change
Could not solve for environment specs
The following package could not be installed
└─ rapids 25.02**  is not installable because it requires
   └─ cuvs 25.02.* , which requires
      └─ __glibc >=2.28,<3.0.a0 , which is missing on the system.

output of 'conda info' (click me)

     active environment : JupyterSystemEnv
    active env location : /home/ec2-user/anaconda3/envs/JupyterSystemEnv
            shell level : 1
       user config file : /home/ec2-user/.condarc
 populated config files : /home/ec2-user/anaconda3/.condarc
                          /home/ec2-user/.condarc
          conda version : 24.11.3
    conda-build version : not installed
         python version : 3.12.8.final.0
                 solver : libmamba (default)
       virtual packages : __archspec=1=broadwell
                          __conda=24.11.3=0
                          __cuda=12.4=0
                          __glibc=2.26=0
                          __linux=5.10.233=0
                          __unix=0=0
       base environment : /home/ec2-user/anaconda3  (writable)
      conda av data dir : /home/ec2-user/anaconda3/etc/conda
  conda av metadata url : None
           channel URLs : https://conda.anaconda.org/conda-forge/linux-64
                          https://conda.anaconda.org/conda-forge/noarch
                          https://conda.anaconda.org/nvidia/linux-64
                          https://conda.anaconda.org/nvidia/noarch
                          https://conda.anaconda.org/pytorch/linux-64
                          https://conda.anaconda.org/pytorch/noarch
          package cache : /home/ec2-user/anaconda3/pkgs
                          /home/ec2-user/.conda/pkgs
       envs directories : /home/ec2-user/anaconda3/envs
                          /home/ec2-user/.conda/envs
               platform : linux-64
             user-agent : conda/24.11.3 requests/2.32.3 CPython/3.12.8 Linux/5.10.233-224.894.amzn2.x86_64 amzn/2 glibc/2.26 solver/libmamba conda-libmamba-solver/24.9.0 libmambapy/1.5.12
                UID:GID : 1000:1000
             netrc file : None
           offline mode : False

Notes

I don't think we'll be able to install RAPIDS 25.02 directly on SageMaker Notebook Instances until SageMaker supports an operating system with at least GLIBC 2.28, like AL2023 (the successor to Amazon Linux 2). From https://docs.aws.amazon.com/linux/al2023/ug/glibc-gcc-and-binutils.html

If just waiting isn't an option, we might also be able to get around this by containerizing all Sagmaker examples here:

• https://docs.aws.amazon.com/sagemaker/latest/dg/studio-byoi-create.html
• https://docs.aws.amazon.com/sagemaker/latest/dg/docker-containers.htm

[jameslamb]

Worth noting: all of the examples using SageMaker Estimators (e.g. for hyperparameter optimization) are already containerized.

So it's really just direct installation on SageMaker notebooks that is affected. Here:

deployment/source/cloud/aws/sagemaker.md

Lines 36 to 55 in e3c471f

	Give your configuration a name like `rapids` and paste the following script into the "start notebook" script.
	```bash
	#!/bin/bash
	set -e
	sudo -u ec2-user -i <<'EOF'
	mamba create -y -n rapids {{ rapids_conda_channels }} {{ rapids_conda_packages }} \
	boto3 \
	ipykernel \
	sagemaker
	conda activate rapids
	python -m ipykernel install --user --name rapids
	echo "kernel install completed"
	EOF
	```