Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Error when Resuming pre-training #67

Open
richardsun-voyager opened this issue May 3, 2024 · 0 comments
Open

Error when Resuming pre-training #67

richardsun-voyager opened this issue May 3, 2024 · 0 comments

Comments

@richardsun-voyager
Copy link

I ran into this error when resuming the pre-training on the pre-trained weights:
"ImportError: cannot import name '_FaultTolerantMode' from 'pytorch_lightning.utilities.enums' "
Has anyone seen this issue before?
Thanks!

I modified the config file by adding the pretrained model path:
train:
gpu_mem: ${eval:"round(float(import('subprocess').check_output('nvidia-smi -i 0 --query-gpu=memory.total --format=csv,noheader,nounits', shell=True).strip().decode()) / 1000)"}
seed: 2222
global_batch_size: 256 # effects the scheduler, need to set properly
pretrained_model_path: 'data/project/hg38/weights.ckpt'

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant