Network Device in Cross-Node Communication Profiling

[li-rongzhi]

Thank you for the helpful profiling data for the simulation. The "send_recv" metric for pipeline parallelism indicates cross-node communication, but I fail to find documentation on the specific network device used for this. Knowing whether the communication happens over Ethernet, InfiniBand, or another device is important for accurately understanding the simulation results, as different devices can significantly impact latency and throughput.
Could you please provide details about the network device used for cross-node communication?

[AgrawalAmey]

Hello Rongzhi,

The network type depends on the cluster type. Here is the mapping:

A100 80GB DGX: 8x200Gbps Infiniband
H100 80GB DGX: 8x400Gbps Infiniband
4xA100 80GB Pairwise NVLink Node: 1x100Gbps Ethernet
8xA40 Pairwise NVLink Node: 1x100Gbps Infiniband

Thanks!