Question about kl_penalty

[StarDewXXX]

kl_penalty is calculated into batch['token_level_reward'] in function apply_kl_penalty() (trainer/ppo/ray_trainer.py). But in function update_policy(), kl_loss is added to the final loss again. (workers/actor/dp_actor.py). So KL penalty might be applied twice?

[PeterSH6]

Hi @StarDewXXX, if kl_loss is used, the kl_penalty will not be applied to the reward. You can see the code: https://github.com/volcengine/verl/blob/main/verl/trainer/ppo/ray_trainer.py#L717-L723