Skip to content

GPU Security

Jump to bottom
Marek Marczykowski-Górecki edited this page Oct 3, 2024 · 2 revisions

Isolation of processes using NVIDIA GPU: summary

Two variants of isolation were investigated:

  1. Virtual machine with GPU passthrough
  2. Docker
  3. Both mechanisms work correctly (allow running CUDA-based code) on all tested GPUs (from Pascal, Turing, Ampere, and Ada generations).

Virtual machine

This mechanism gives full access to the GPU to the guest machine at the PCI Express bus level. The IOMMU mechanism protects the host from memory access by the GPU during the operation of the guest machine. However, we consider the following attacks:

  1. Whole machine suspension by the guest (DoS attack)
  2. Damage to the GPU (e.g., by disabling mechanisms protecting against overheating)
  3. Modification of the GPU VBIOS

The first of these attacks turns out to be easy to carry out on each of the tested GPUs by writing to the PCI Express bus control registers located in BAR0. It is possible to disable the GPU's response to PCIE requests, preventing further communication until the machine is reset. On one of the tested machines, this state led to a soft lockup in the host kernel, and on the other machine, it immediately suspended the entire host machine.

Modern NVIDIA GPUs (from the Maxwell generation upwards) protect against the remaining two attacks through a complex system of access control to the BAR0 registers, limiting access to sensitive registers only to trusted firmware running on the GPU itself. This firmware is verified by a cryptographic signature by the immutable bootROM in the GPU.

The complexity of this mechanism, the lack of any documentation of the interesting parts of the hardware, and the fact that the firmware itself is encrypted, make it impossible to carry out an effective security audit in a reasonable time. It should also be mentioned that slightly older versions of this mechanism have been effectively broken (https://github.com/CAmadeus/falcon-tools).

Finding a loophole in these protections and the possibility of arbitrary modification of VBIOS would mean a complete breakdown of isolation - the modified VBIOS could run firmware on the card that performs DMA attacks on the host machine or other guest machines. It is also quite likely that any loopholes found will be at the hardware level and therefore impossible to patch.

Moreover, NVIDIA does not support the use of GPU passthrough on consumer hardware - officially, this is only allowed on selected data center type cards, on a supported hypervisor, and after purchasing an additional license. If bugs are found, the chances of receiving a patch will be slim.

Docker

Docker's isolation relies on operating system namespace mechanisms. As for the GPU itself, NVIDIA does not have any special Docker isolation mechanisms and from a security perspective, using a GPU in a Docker container (or similar) is equivalent to using a GPU by an ordinary, non-privileged user.

The attack surface available to the GPU user is huge and consists of:

Kernel interface accessible via /dev/nvidia* files (part of this interface is documented here: https://github.com/NVIDIA/open-gpu-kernel-modules/tree/main/src/common/sdk/nvidia/inc/ctrl/); most of these interfaces are only accessible to processes with CAP_SYS_ADMIN Firmware modules handling user commands (FECS, NVDEC, NVENC, OFA, NVJPG) Hardware interfaces directly controlled by the user Again, conducting an effective audit is not possible here.

The situation here is, however, in some respects better than in the case of passthrough: this use case is fully supported by the manufacturer, and any security holes will almost certainly be fixable through driver updates.

The isolation provided by Docker itself, however, is much weaker than in the case of a virtual machine - bugs allowing escape from Docker occur regularly. A recent example affecting Nvidia Container Toolkit specifically: https://www.wiz.io/blog/wiz-research-critical-nvidia-ai-vulnerability

Recommendations

In our assessment, none of the investigated mechanisms provide security guarantees. Where possible, we recommend the implementation of a defense-in-depth strategy and the combination of several mechanisms:

  • Limiting GPU instances to trusted requestors
  • Using a virtual machine with GPU passthrough
  • Running a trusted (and regularly updated) kernel and driver in the virtual machine
  • Using further isolation within the virtual machine based on namespace and capabilities mechanisms (in particular, ensuring that the workload does not have access to CAP_SYS_ADMIN)
Clone this wiki locally