Skip to content

Commit

Permalink
add barrier before process group cleanup (#1048)
Browse files Browse the repository at this point in the history
iiuc not necessary because our ops are all sync but good to include in case users take this code and run more complex scripts with it
  • Loading branch information
charlesfrye authored Jan 16, 2025
1 parent b8b6a83 commit af3b503
Showing 1 changed file with 1 addition and 0 deletions.
1 change: 1 addition & 0 deletions 14_clusters/simple_torch_cluster_script.py
Original file line number Diff line number Diff line change
Expand Up @@ -87,6 +87,7 @@ def init_processes(backend):
dist.init_process_group(backend, rank=WORLD_RANK, world_size=WORLD_SIZE)
yield
finally:
dist.barrier() # ensure any async work is done before cleaning up
# Remove this if it causes program to hang. ref: https://github.com/pytorch/pytorch/issues/75097.
dist.destroy_process_group()

Expand Down

0 comments on commit af3b503

Please sign in to comment.