Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

The effect is worse after fine-tuning #188

Open
liyujia011025 opened this issue Feb 16, 2025 · 0 comments
Open

The effect is worse after fine-tuning #188

liyujia011025 opened this issue Feb 16, 2025 · 0 comments
Labels
bug Something isn't working

Comments

@liyujia011025
Copy link

Hello expert:
I used my own data to fine tune the Moirai large model according to the fine-tuning process and code in the official README file, and found that the effect after fine-tuning was actually worse than without fine-tuning, which is strange.
Among them, because I set the context length to 96 and the prediction length to 4 during prediction, I set the parameters in the cli/conf/finetune/val_data/data.yaml file as shown in the figure. The learning rate in moirai-1.0-R-small.yaml was set to 1e-7, and other places such as hyperparameters were not changed. I tried fine-tuning all layers of Moirai and some layers of the output layer separately, but the results were even worse.
Do you know where my mistake occurred? Do I need to make specific changes to the hyperparameter settings or other content during the fine-tuning process?
Looking forward to your reply, thank you very much!

Image

@liyujia011025 liyujia011025 added the bug Something isn't working label Feb 16, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

1 participant