Use of APIs for GRPO reward ?

[ocramz]

Thinking of verifiable reward models: we have a program verifier that would accept Python program strings as well as pre-and post-condition predicates, and return execution traces as well as counterexamples.

We are exploring the possibility of opening up this functionality as a REST API, as a simpler integration path than reimplementing this functionality in Python for the open-r1 repo.

Questions:

• Is this kind of API integration compatible with the openness aim of open-r1?
• Does the GRPOTrainer cache function invocations?
• Related: would GRPOTrainer build a list of prompt-answer pairs therefore a dataset out of the API invocations?

Thanks!

[Muhtasham]

Seems like could be done on Modal Sandbox

[ocramz]

Here's a PoC for what's described above! Still some details to iron out namely prompting format, but the foundations are there: #376