Support NVIDIA MIG for more secure multi-tenant GPU sharing / allocation

[jcmcken]

What I'd like:

NVIDIA time-slicing landed (see #2347) in Bottlerocket 1.25. While a step forward, this can't really be used all that securely in a multi-tenant EKS cluster (as mentioned in the documentation).

NVIDIA MIG provides a more secure alternative.

Here are some references:

• https://www.infracloud.io/blogs/gpu-sharing-techniques-guide-vgpu-mig-time-slicing/#vgpu-vs-mig-vs-gpu-timeslicing
• https://docs.nvidia.com/datacenter/cloud-native/kubernetes/latest/index.html#mig-support-in-kubernetes
• https://github.com/awslabs/aws-virtual-gpu-device-plugin
• https://aws.amazon.com/blogs/containers/utilizing-nvidia-multi-instance-gpu-mig-in-amazon-ec2-p4d-instances-on-amazon-elastic-kubernetes-service-eks/

Any alternatives you've considered:

NVIDIA vGPU is another alternative, but it has fewer isolation guarantees than MIG. It's also not all that clear how it would be used in EKS, and I think might require a commercial license.

Today, in our multi-tenant clusters, the only solution seems to be that each pod receives its own GPU worker node, which is highly cost ineffective.

[piyush-jena]

Hi @jcmcken ! MIG is a prioritized feature on the Bottlerocket roadmap, and the team is actively working on its development. While we cannot commit to a specific delivery date, we aim to provide an update on the progress in the next two weeks. Please use this issue for the tracking purposes. Thank you for your interest.