Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Is it suitable to calculate busBw in AllReduceGetBw for all of the algos? #279

Open
shanleo2024 opened this issue Jan 9, 2025 · 0 comments

Comments

@shanleo2024
Copy link

Hi Dear developer,
In AllReduceGetBw, the busBw = baseBw * ((double)(2*(nranks - 1)))/((double)nranks)

I think it is good for ring algorithm, but for the other algorithm, such as Tree/NVLS, does is suitable to use (2*(nranks - 1)))/((double)nranks?
As from the source code of NCCL, the ratio of converting bus BW to algorithm BW is different.

        // Convert bus BW to algorithm BW
        float ratio;
        if (a == NCCL_ALGO_RING) ratio = (1.0 * nRanks) / nsteps;
        else if (a == NCCL_ALGO_NVLS) ratio = 5.0/6.0;
        else if (a == NCCL_ALGO_NVLS_TREE) ratio = .70 * nNodes / (2*(nNodes-1));
        else ratio = .5;
        comm->bandwidths[coll][a][p] = busBw * ratio;

Thank you.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant