⚡ Fix GRPO PEFT

[qgallouedec]

What does this PR do?

Fixes # (issue)

Before submitting

• This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
• Did you read the contributor guideline,
  Pull Request section?
• Was this discussed/approved via a GitHub issue? Please add a link
  to it if that's the case.
• Did you make sure to update the documentation with your changes? Here are the
  documentation guidelines.
• Did you write any new necessary tests?

Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

[tgaddair]

Hey @qgallouedec, just want to flag that this is quite a compute and memory intensive operation, and there may be a better way to do this.

Assuming the underlying Peft model is LoRA, vLLM already supports native dynamic LoRA inference, which can be done like so:

self.llm = LLM(
            ...,
            enable_lora=True,
        )

lora_request = LoRARequest(
     "<unique_str>",
     self.state.global_step,  # needs to be a unique identifier
     lora_path=checkpoint_dir_containing_adapter_weights,
)

outputs = self.llm.generate(
    all_prompts_text,
    sampling_params=self.sampling_params,
    lora_request=lora_request,
    use_tqdm=False
)

[qgallouedec]

Wow thanks you so much for the pointer. Let me try this

[gagan3012]

Hello,
Are there any updates on this PR as this is currently a blocker in my work.

[qgallouedec]

It's WIP, high priority. But if you find a fix in the meantime feel free to controbute 😊

[tgaddair]

Hey @qgallouedec, please see #2730 for a follow-up that implements my suggestion.