openrlhf-training
High-performance RLHF framework with Ray+vLLM acceleration. Use for PPO, GRPO, RLOO, DPO training of large models (7B-70B+). Built on Ray, vLLM, ZeRO-3. 2× faster than DeepSpeedChat with distributed architecture and GPU resource sharing.
npx skills add Orchestra-Research/AI-Research-SKILLs --skill openrlhf --agent claude-code
Same command for any agent — swap --agent for codex, cursor, copilot.
Weekly change comes from our own snapshots, not the repository page — it measures attention, not adoption.
# OpenRLHF - High-Performance RLHF Training ## Quick start OpenRLHF is a Ray-based RLHF framework optimized for distributed training with vLLM inference acceleration. **Installation**: ```bash # Launch Docker container docker run --runtime=nvidia -it --rm --shm-size="10g" --cap-add=SYS_ADMIN \ -v $PWD:/openrlhf nvcr.io/nvidia/pytorch:25.02-py3 bash # Uninstall conflicts sudo pip uninstall xgboost transformer_engine flash_attn pynvml -y # Install OpenRLHF with vLLM pip install openrlhf[vllm] ``` **PPO Training** (Hybrid Engine): ```bash ray start --head --node-ip-address 0.0.0.0 --num-gpus 8 ray job submit --address="http://127.0.0.1:8265" \ --runtime-env-json='{"working_dir": "/openrlhf"}' \ -- python3 -m openrlhf.cli.train_ppo_ray \ --ref_num_nodes 1 --ref_num_gpus_per_node 8 \ --reward_num_nodes 1 --reward_num_gpus_per_node 8 \ --critic_num_nodes 1 --critic_num_gpus_per_node 8 \ --actor_num_nodes 1 --actor_num_gpus_per_node 8 \ --vllm_num_engines 4 --vllm_tensor_parallel_size 2 \ --colocate_all_models \ --vllm_gpu_memory_utilization 0.5 \ --pretrain OpenRLHF/Llama-3-8b-sft-mixture \ --reward_pretrain OpenRLHF/Llama-3-8b-rm-700k \ --save_path ./output/llama3-8b-rlhf \ --micro_trai
- Quick start
- Common workflows
- Workflow 1: Full RLHF pipeline (SFT → Reward Model → PPO)
- Workflow 2: GRPO training (no critic model needed)
- Workflow 3: DPO training (preference optimization)
- When to use vs alternatives
- Common issues
- Advanced topics
- Hardware requirements
- Resources
Launch Docker container docker run --runtime=nvidia -it --rm --shm-size="10g" --cap-add=SYS_ADMIN \ Uninstall conflicts sudo pip uninstall xgboost transformer_engine flash_attn pynvml -y Install OpenRLHF with vLLM pip install openrlhf[vllm] ray start --head --node-ip-address 0.0.0.0 --num-gpus 8 ray job submit --address="http://127.0.0.1:8265" \ Same command as PPO, but add: deepspeed --module openrlhf.cli.train_rm \
What does the openrlhf-training skill do?
High-performance RLHF framework with Ray+vLLM acceleration. Use for PPO, GRPO, RLOO, DPO training of large models (7B-70B+). Built on Ray, vLLM, ZeRO-3. 2× faster than DeepSpeedChat with distributed architecture and GPU resource sharing.
How do I install it?
Run `npx skills add Orchestra-Research/AI-Research-SKILLs --skill openrlhf --agent claude-code` — it drops the skill into your project so the agent can pick it up. Swap the --agent value for codex, cursor or copilot if you use one of those.
Where does this skill come from?
From Orchestra-Research/AI-Research-SKILLs, a repository with 11,391 stars. We read it straight from the repository tree rather than a submitted listing, so what you see here is what is actually published.
Is a popular skill a good skill?
Not necessarily. Stars measure attention, not adoption — a repository can trend for a week and be abandoned. That is why we show the weekly change from our own snapshots next to the total, instead of a single flattering number.