Question about reward function in GRPO example

[junuMoon]

https://huggingface.co/docs/trl/main/en/grpo_trainer

Am I misunderstanding something about the reward function in the documentation example?

Looking at the current example:

def reward_len(completions, **kwargs):
    return [abs(20 - len(completion)) for completion in completions]

I think this reward function might be working opposite to what we want. Since GRPO tries to maximize the reward, wouldn't this make the model generate text that's far from 20 characters?

Shouldn't we use something like:

def reward_len(completions, **kwargs):
    return [-abs(20 - len(completion)) for completion in completions]

or

def reward_len(completions, **kwargs):
    return [1 / (1 + abs(20 - len(completion))) for completion in completions]

so that completions closer to 20 characters get higher rewards?

Would appreciate if someone could clarify this

[qgallouedec]

Ah yes, you are right. Do you want to correct it?

[cfpark00]

#2714

[cfpark00]

will close this as well when addressed!

[junuMoon]

Is it okay that I make PR on this?

[susumuota]

Thank you so much! I have the same problem.

I'm trying this reward function.

def reward_len(completions, **kwargs):
    return [np.exp(-0.1 * abs(len(completion) - 20)) for completion in completions]