Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[PT-Inductor:GPU] Out of memory on BERT_LARGE FP32 with large batch sizes #61

Open
pzread opened this issue Jul 7, 2023 · 1 comment
Open

[PT-Inductor:GPU] Out of memory on BERT_LARGE FP32 with large batch sizes #61

pzread opened this issue Jul 7, 2023 · 1 comment
Labels
bug Something isn't working

Comments

@pzread
Copy link

pzread commented Jul 7, 2023

Large batch sizes (>=64) on BERT_LARGE FP32 runs out of CUDA memeory on a2-highgpu-1g

https://github.com/openxla/openxla-benchmark/actions/runs/5483405564/jobs/9989714567#step:5:923
@pzread pzread changed the title [PT-Inductor:GPU] Out of memory on BERT_LARGE with large batch sizes [PT-Inductor:GPU] Out of memory on BERT_LARGE FP32 with large batch sizes Jul 7, 2023
@pzread pzread mentioned this issue Jul 7, 2023
Merged
@pzread
Copy link
Author

pzread commented Jul 10, 2023

Same issue also happened to ResNet50 with large batch sizes: https://github.com/openxla/openxla-benchmark/actions/runs/5502589911/jobs/10026944617#step:5:851

@pzread pzread mentioned this issue Jul 10, 2023
Merged
@pzread pzread added the bug Something isn't working label Aug 8, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@pzread