Task-Specific GPU Requirement to Improve Resource Utilization

[alonkukl]

The current runtime environment distributes tasks evenly among workers, regardless of whether they require access to GPUs. This leads to inefficiencies, particularly in scenarios where only a small subset of tasks actually require GPUs. Moreover, if a single task requires a GPU, then all workers must have GPU access, which is inefficient in environments where a server may have over 100 workers but only a handful of GPUs.

I propose introducing a mechanism to specify whether a particular task/pass requires GPU resources. This enhancement would allow the scheduler to distribute them more intelligently, ensuring that GPU-bound tasks are assigned only to workers with access to GPUs, while CPU-bound tasks are distributed to non-GPU workers. This would eliminate the current inefficiency of needing to allocate all tasks to GPU-equipped workers when only a some of them need GPU resources.

Suggested Implementation

• Add a task-specific flag to indicate GPU requirements.
• Add a flag to the worker to indicate if it has access to a GPU or not.
• Modify the task scheduler to distribute GPU-required tasks exclusively to GPU-equipped workers and non-GPU tasks to CPU-only workers.

Alternative Workaround

One might have two runtime environments, one with GPUs and the other without, and the task would be submitted to the appropriate runtime.