Can we use VeRL to train the reward models

[YSLIU627]

Hi Authors,
Your repo is really good and I wonder if we could use VeRL to train the reward models just like OpenRLHF.

[vermouth1992]

Actually, we have a DPO implementation already on a branch. Training ORM should be pretty similar to training DPO. Do you also need training PRM?

[YSLIU627]

No, I only need to train the ORM. However, the DPO implementation requires the same number of positive and negative samples, while ORM training in math can tolerate such an imbalance. It would help a lot if you could have a script for the ORM training!