Getting Exponentially large losses for CLIFF after second epoch

[ShamLich]

I attempted to follow the convert datasets code for CLIFF, but there are some issues when I try to train using those datasets, I get exponentially large MPJPE, and PA-MPJPE losses. It would be really helpful!

Example on 9th epoch:
2023-05-15 11:38:16,965 - mmhuman3d - INFO - Epoch [9][2000/4878] lr_backbone: 3.000e-04 lr_head: 3.000e-04, eta: 0:00:00, time: 1.417, data_time: 0.030, memory: 3198, keypoints3d_loss: 1079724.9350, keypoints2d_loss: 130367.2165, vertex_loss: 18867.5181, smpl_pose_loss: 2.0817, smpl_betas_loss: 9921199272.9600, loss: 9922428395.5200