Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[skip lazy allocation feature] add comm part to et_replay without lazy allocation of tensors #177

Conversation

TaekyungHeo
Copy link
Contributor

@TaekyungHeo TaekyungHeo commented Aug 30, 2024

Test Plan

#!/bin/bash
#SBATCH --job-name=full_replay_job
#SBATCH --nodes=1
#SBATCH --ntasks=8
#SBATCH --ntasks-per-node=8
#SBATCH --gpus-per-node=8
#SBATCH --time=01:00:00
#SBATCH --output=%x_%j.out
#SBATCH --error=%x_%j.err


srun -A general_cs_plutus\
     --container-image /lustre/fsw/portfolios/general/users/songyant/docker/songyant+docker-images+ai_develop.sqsh \
     --no-container-mount-home \
     --container-mounts=/lustre/fsw/portfolios/general/users/songyant/workspace_local:/workspace_local \
     zsh -c "

        if [ ! -d /run/sshd ]; then
            mkdir -p /run/sshd
            chmod 0755 /run/sshd
        fi

        source ~/.zshrc
        conda activate full_replay

        et_replay --trace_path /workspace_local/full_replay/traces/resnet-8gpu
     "

image

@facebook-github-bot facebook-github-bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Aug 30, 2024
root added 2 commits September 3, 2024 20:49
…ll cause the replay hang, so add an option to config env vars when run local without slurm or mpirun
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants