[Question] Serving Llama 3.3 70B model with 8x RTX 4090 GPUs using Triton Inference Server without NVLink

[novela77]

Hello,

I am planning to serve the Llama 3.3 70B model using Triton Inference Server with the TensorRT-LLM backend on a system equipped with 8 NVIDIA RTX 4090 GPUs. My questions are:

1. Is it feasible to serve the Llama 3.3 70B model in FP16 precision under this configuration?
2. Can this setup operate efficiently without NVLink, or would the absence of NVLink significantly impact performance?
3. Are there any additional considerations or recommendations for optimizing performance in this environment?

Thank you for your assistance.