Code

Bugfixes: Grad norm scaling in TP#2436

Code

Draft

nathan-az wants to merge 6 commits intopytorch:mainpytorch/torchtune:mainfrom nathan-az:grad-norm-replicationnathan-az/torchtune:grad-norm-replicationCopy head branch name to clipboard

+32-13

Commits

Commits on Feb 25, 2025

• Use implicit replication in distributed train step to fix grad norm clipping

  Show description for 0688528
  nathan-az

  committed
  0688528

  Copy full SHA for 0688528

• calculate and report grad norm even if not clipping, fix tok/sec reporting

  Show description for 8f4c734
  nathan-az

  committed
  8f4c734

  Copy full SHA for 8f4c734

• fix foreach typo

  Show description for 136d1ce
  nathan-az

  committed
  136d1ce

  Copy full SHA for 136d1ce

• Adjust grad norm scaling and tps counter

  Show description for 8d0c044
  nathan-az

  committed
  8d0c044

  Copy full SHA for 8d0c044

• Inline passing params to grad norm calc

  nathan-az

  authored
  d498a58

  Copy full SHA for d498a58

• Add None filter

  nathan-az

  authored
  71daab1

  Copy full SHA for 71daab1