-
Notifications
You must be signed in to change notification settings - Fork 64
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
EmptyLauncher supported cuda_visible_devices by memory left order #305
Conversation
lazyllm/launcher.py
Outdated
gpus = self.launcher._get_idle_gpus() | ||
if gpus: | ||
if self.launcher.ngpus in (None, 0): | ||
empty_cmd = f'export CUDA_VISIBLE_DEVICES={gpus[0]} && ' |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
不要export
@@ -173,6 +174,23 @@ class Job(Job): | |||
def __init__(self, cmd, launcher, *, sync=True): | |||
super(__class__, self).__init__(cmd, launcher, sync=sync) | |||
|
|||
def _wrap_cmd(self, cmd): | |||
gpus = self.launcher._get_idle_gpus() |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
可以先判断 self.launcher.ngpus,如果为0,则直接跳过,否则再_get_idle_gpus
lazyllm/launcher.py
Outdated
gpus = self.launcher._get_idle_gpus() | ||
if gpus and lazyllm.config['cuda_visible']: | ||
if self.launcher.ngpus in (None, 0): | ||
empty_cmd = f'CUDA_VISIBLE_DEVICES={gpus[0]} && ' |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
这里不要&&,empty应该是没问题的
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
slurm应该也没问题,sensecore不确定
EmptyLauncher supports setting
CUDA_VISIBLE_DEVICES
based on the remaining GPU memory for each GPU.