Multires hubert #5363
Multires hubert #5363
Conversation
| untie_final_proj: true | ||
| activation_dropout: 0.0 | ||
| conv_adapator_kernal: 1 | ||
| use_single_target: true |
There was a problem hiding this comment.
Hi @ftshijt, first thank you for this interesting research and the pre-training scripts. If I made it right, according to the paper (Appendix B.4), predicting a single target is proved a little bit worse than doing multi-task, why in all the configs it is set to true please?
There was a problem hiding this comment.
Thanks for the review!
This argument has a different meaning: it refers to using the same discrete token sequence for both low-resolution and high-resolution (while the low-resolution is generated by skip-downsampling). We also support using different tokens (refer to Appendix B.7).
For multi-task vs. single-task, we have a "use_single_prediction" argument for that and the default is false, so that we can focus on using multi-task.
There was a problem hiding this comment.
But thanks for pointing out the confusion, I will later update a few notes in the README to make it clear to future users.
There was a problem hiding this comment.
Thank you, now it is more clear to me. I was just confused by the term, will read the code thoroughly.
|
Many thanks for the code review! @annasun28 |
Before submitting
What does this PR do?
Adding new multiresolution HuBERT implementation to fairseq https://arxiv.org/abs/2310.02720
PR review
Anyone in the community is free to review the PR once the tests have passed.
If we didn't discuss your PR in Github issues there's a high chance it will not be merged.
TODOs (in progress)