-
Notifications
You must be signed in to change notification settings - Fork 1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
RuntimeError: Error(s) in loading state_dict for EmbeddingPipe: size mismatch for word_embeddings.weight #645
Comments
I am getting the same problem too when trying to train a 1-3B model. To Reproduce:
Environment:
|
What's the solution ? and why closed ? |
@djaym7 Thanks for saying something. I don't recall closing this and have reopened it. |
@FayZ676 the url you’re linking to does not contain the weights for a 1.3B model, it contains the weights for a 20B model. That’s why you’re getting a size mismatch: it’s quite simply the wrong size. I suspect that this is unrelated to the problems the others are having. @leclem so that change allows you to finetune the 20B model? Can you post a WandB link showing it training so I can check out the loss etc are as expected? |
I have the same issue trying to train. Downloaded slim weight and using ./config/20B.yml and running "python3 ./deepy.py train.py ./configs/20B.yml" gives this error: RuntimeError: Error(s) in loading state_dict for EmbeddingPipe: |
I suspect that this is an error that has to do with model parallelism. @shaunstoltz how many GPUs were you loading the model onto / what was the model parallelism setting? |
Does anyone have a solution for this problem? |
Describe the bug
RuntimeError: Error(s) in loading state_dict for EmbeddingPipe: size mismatch for word_embeddings.weight: copying a param with shape torch.Size([25216, 6144]) from checkpoint, the shape in current model is torch.Size([50304, 6144]).
To Reproduce
./configs/20B.yml
(HFTokenizer is used)./deepy.py generate.py ./configs/20B.yml -i prompt.txt -o sample_outputs.txt
Screenshots
Environment (please complete the following information):
The text was updated successfully, but these errors were encountered: