does DRA support multi GPUs across worker nodes?

[thj08]

I want to let master node have ability to allocate avaliable GPUs across different worker nodes , does DRA support multi GPUs across worker nodes?

[ArangoGutierrez]

So far DRA enables resources in a per-Node scenario given the interaction with the Kubelet. What you are asking is a MultiNode DRA if I understand correctly

[thj08]

Thanks for the reply.
As my real case, I setup my cluster with 1 master node and 3 worker nodes, and every worker node has one GPU resource. Is it possible to apply a container on one worker node with 3 GPUs?

By the way, I also try the DRA demo project, and it use kind for local cluster. Does DRA support remote cluster?

[asm582]

I think you are talking about DRA and CXL integration. This is discussed as one of the use case but currently not implemented as I understand.

[klueska]

Sorry for the late reply. I don't quite understand your question. You want a container running on one node to make use of both its 1 local GPU as well as the two remote GPUs (but make them appear to the application as if they are local)?

That is not something that is possible, nor something that we plan to support directly.

In the (near) future we will start to support the allocation of IMEX channels to allow GPUs on one node to read/write the GPU memory of other nodes:
https://docs.nvidia.com/cuda/cuda-runtime-api/group__CUDART__MEMORY__POOLS.html#group__CUDART__MEMORY__POOLS_1g8158cc4b2c0d2c2c771f9d1af3cf386e

But even this requires you to still run a separate container on each node and implement the application logic to orchestrate these read/writes yourself (or via a library such as NCCL).