PRIME is a Python-based scalable RL framework for advanced reasoning in LLMs, focusing on implicit process reward modeling and online RL with PPO. It provides training, data preprocessing, and evaluation components, with documentation links and usage examples.
Collecting history — the radar snapshots this repo daily. The trend line appears after 3 days of data (1 so far).
What it is
Process Reinforcement Through Implicit Rewards (PRIME) is a scalable RL solution intended to improve reasoning abilities of large language models by using implicit process rewards (PRM) and an outcome verifier. The project describes online RL training where a policy model and a PRM are initialized from a base SFT model, and rollouts are updated with implicit process rewards and outcome rewards. The README states PRIME integrates with veRL and uses vLLM for inference, with a focus on dense reward signals per token and on-policy updates.
How it works
The approach involves generating rollouts with a policy model, scoring them with an implicit PRM and outcome verifier, updating the implicit PRM based on predicted process rewards and ground truth outcomes, and then combining the outcome reward r_o and process reward r_p to update the policy via PPO loss. The evaluation uses leave-one-out baselines for process rewards and discounted returns, with RLOO applied to outcome rewards. The algorithm flow includes prompt filtering, computing r^t, updating the implicit PRM, calculating combined advantages, and updating the policy.
Getting started
Getting started information is provided via sections in the README: Getting Started lists directories for PRIME-related code: training, data_preprocessing, and eval. It also references an external repository for ImplicitPRM training and evaluation. Usage examples include prompt formats for Coding and Math tasks, and an embedded Python inference snippet showing how to configure an LLM for generation. The prompts and code blocks are shown verbatim in the README.
Recent releases
RELEASES (latest 0) indicates there are no releases listed. The README includes a News section with dates 2025/03/12, 2025/02/04, 2025/01/06, and 2025/01/02 detailing integration, arXiv paper release, preprocessing code, and announcement of PRIME.
Traction
Stars: 1869. Forks: 116. Open issues: 10. Language: Python. License: Apache-2.0. Created: 2024-12-31. Last push: 2025-03-18. Topic tags include llm, reasoning, rl. The star history chart is referenced but no numeric traction metrics beyond the star count are given in the README.
Caveats
License is Apache-2.0. Creation date 2024-12-31 and last push 2025-03-18 are provided in the facts. The README mentions integration and external repositories (veRL, vLLM, Eurus, Qwen2.5-Math) and datasets, but no explicit open issues or known caveats are listed beyond the general content. No explicit installation commands are provided in the extracted README content here beyond usage sections and directory references.