Training strategy for Zipformer using fp16 ??

[ZQuang2202]

Hi everyone,
I am a student attempting to reproduce the results of Zipformer on Librispeech 100h, facing limitations in hardware resources that prevent me from using the recommended configuration. Due to these constraints, I have reduced the batch size (max_duration) to 300, as opposed to the recommended 1000. However, I am struggling to find the appropriate configuration for Eden.

1. Following the training strategy that suggests decreasing the learning rate by √k times when the batch size decreases by k times, I initially set the base_lr to 0.03, keeping other configurations at their default values. But the training process diverges. Despite attempts to adjust lr_batches, lr_epochs (3.5-6), and base_lr (0.3-0.45), it's still not working. Notably, the training process encounters divergence when the batch_count is around 700-900, leading to 'parameter domination' issues in the embed_conv and some attention modules. I attach some log information below.
   
   
   

2. In an effort to address this, I attempted to reduce the gradient scale of the layers experiencing 'parameter domination,' but this proved ineffective."

I have few questions:

1. Is there some thing related between base_lr/lr_batch/lr_epoch and max_duration?
2. Should parameters like dropout rate or min_abs, max_abs of Balancer or Whitener's config be changed? If so, how should it be changed for each different dataset?
   Thank you.

[yaozengwei]

Are you using single GPU and max-duration=300? The gradient noise might be large with such a small batch size. You could try a smaller base-lr, like 0.025, and keep lr_batch/lr_epoch unchanged. Usually you don't need to tune the Balancer and Whitener configurations.

[ZQuang2202]

Thank you for your response @yaozengwei.
"Are you using single GPU and max-duration=300?" - Yes, I'm using a single RTX 3060 and max-duration=300. I tried base_lr=0.025 but it didn't work. Then, I try base_lr=0.015, and the training process converges. However, the convergence is quite slow. Besides the hyperparameter of Eden, are there any important parameters I need to consider?