how overall throughout calculate about all2all

[ltm920716]

Hi，
I have six H100 nodes，and each with 8*400Gb cx7 nics. And for RDMA, I use RoCE. I want to see the overall throughout.

about allreduce, it seems that the params effect little，and the busbw is the overall throughout？

abou all2all，the params effect large，as follows：

and for all2all，the busbw is for single node or something else？How can I calculate the overall throughout？I can not understand deeply about the busbw for all2all，and what params are the best to test alltoall？the performence will down with the same config when add more node

thanks！

[sjeaugey]

All numbers look good. Alltoall cannot aggregate the bandwidth of multiple NICs, so the performance you should see is the performance of a single NIC.

[ltm920716]

so how should I calculate the single nic performance？or could you recommend me a link to understand the calculation formula？

[joydchh]

All numbers look good. Alltoall cannot aggregate the bandwidth of multiple NICs, so the performance you should see is the performance of a single NIC.

i have some questions here, why the busbw can exceed 50GB/s? it's also related to sharp or something else?

[AddyLaddy]

CX-7 NICs at NDR rate (400Gbps) should achieve up to 48-49GB/s per NIC.
The BW reported for an A2A will be the average over all the P2P communications, and some of those will be over NVLink at around 350GB/s, so these will cause the average to be reported as above the physical speed of the NDR NIC.
You can force all communication via the NICs by setting NCCL_P2P_DISABLE=1 NCCL_SHM_DISABLE=1 but that will also hurt A2A performance as it will disable PXN.

NVLink SHARP is only relevant to where we offload Reduction/Multicast operations to the NVSwitches such as for NCCL AllReduce.

[joydchh]

CX-7 NICs at NDR rate (400Gbps) should achieve up to 48-49GB/s per NIC. The BW reported for an A2A will be the average over all the P2P communications, and some of those will be over NVLink at around 350GB/s, so these will cause the average to be reported as above the physical speed of the NDR NIC. You can force all communication via the NICs by setting NCCL_P2P_DISABLE=1 NCCL_SHM_DISABLE=1 but that will also hurt A2A performance as it will disable PXN.

NVLink SHARP is only relevant to where we offload Reduction/Multicast operations to the NVSwitches such as for NCCL AllReduce.

OK, got it. Thanks for the detailed explanation!