New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Needel_test CUDA OOM 了应该怎么解决？ #11

Open

SefaZeng opened this issue Jun 11, 2024 · 1 comment

SefaZeng commented Jun 11, 2024

token 太多OOM了应该怎么解决？

Member

bys0318 commented Jun 12, 2024

如果条件允许的话，可以用多gpu推理，只需要在load模型时传入device_map="auto"

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment