simpo-training
Simple Preference Optimization for LLM alignment. Reference-free alternative to DPO with better performance (+6.4 points on AlpacaEval 2.0). No reference model needed, more efficient than DPO. Use for preference alignment when want simpler, faster training than DPO/PPO.
npx skills add Orchestra-Research/AI-Research-SKILLs --skill simpo --agent claude-code
Same command for any agent — swap --agent for codex, cursor, copilot.
Weekly change comes from our own snapshots, not the repository page — it measures attention, not adoption.
# SimPO - Simple Preference Optimization ## Quick start SimPO is a reference-free preference optimization method that outperforms DPO without needing a reference model. **Installation**: ```bash # Create environment conda create -n simpo python=3.10 && conda activate simpo # Install PyTorch 2.2.2 # Visit: https://pytorch.org/get-started/locally/ # Install alignment-handbook git clone https://github.com/huggingface/alignment-handbook.git cd alignment-handbook python -m pip install . # Install Flash Attention 2 python -m pip install flash-attn --no-build-isolation ``` **Training** (Mistral 7B): ```bash ACCELERATE_LOG_LEVEL=info accelerate launch \ --config_file accelerate_configs/deepspeed_zero3.yaml \ scripts/run_simpo.py \ training_configs/mistral-7b-base-simpo.yaml ``` ## Common workflows ### Workflow 1: Train from base model (Mistral 7B) **Config** (`mistral-7b-base-simpo.yaml`): ```yaml # Model model_name_or_path: mistralai/Mistral-7B-v0.1 torch_dtype: bfloat16 # Dataset dataset_mixer: HuggingFaceH4/ultrafeedback_binarized: 1.0 dataset_splits: - train_prefs - test_prefs # SimPO hyperparameters beta: 2.0 # Reward scaling (2.0-10.0) gamma_beta_ratio: 0.5 # Target margin (0-1) loss
- Quick start
- Common workflows
- Workflow 1: Train from base model (Mistral 7B)
- Workflow 2: Fine-tune instruct model (Llama 3 8B)
- Workflow 3: Reasoning-intensive tasks (lower LR)
- When to use vs alternatives
- Common issues
- Advanced topics
- Hardware requirements
- Resources
Create environment conda create -n simpo python=3.10 && conda activate simpo Install PyTorch 2.2.2 Install alignment-handbook git clone https://github.com/huggingface/alignment-handbook.git cd alignment-handbook python -m pip install . Install Flash Attention 2 python -m pip install flash-attn --no-build-isolation scripts/run_simpo.py \
What does the simpo-training skill do?
Simple Preference Optimization for LLM alignment. Reference-free alternative to DPO with better performance (+6.4 points on AlpacaEval 2.0). No reference model needed, more efficient than DPO. Use for preference alignment when want simpler, faster training than DPO/PPO.
How do I install it?
Run `npx skills add Orchestra-Research/AI-Research-SKILLs --skill simpo --agent claude-code` — it drops the skill into your project so the agent can pick it up. Swap the --agent value for codex, cursor or copilot if you use one of those.
Where does this skill come from?
From Orchestra-Research/AI-Research-SKILLs, a repository with 11,391 stars. We read it straight from the repository tree rather than a submitted listing, so what you see here is what is actually published.
Is a popular skill a good skill?
Not necessarily. Stars measure attention, not adoption — a repository can trend for a week and be abandoned. That is why we show the weekly change from our own snapshots next to the total, instead of a single flattering number.