unsloth-dpo
Direct Preference Optimization (DPO) in Unsloth provides a way to align models with human preferences using paired data (chosen/rejected). Unsloth optimizes this process by allowing refmodel=None, significantly reducing memory requirements while maintaining high performance.
npx skills add majiayu000/claude-skill-registry --skill unsloth-dpo-cuba6112-skillfactory-f41c8835 --agent claude-code
Same command for any agent — swap --agent for codex, cursor, copilot.
Weekly change comes from our own snapshots, not the repository page — it measures attention, not adoption.
--- name: unsloth-dpo description: Direct Preference Optimization (DPO) for aligning models with preference data without separate reward models. Triggers: dpo, preference optimization, rlhf, ref_model=none, patchdpotrainer, dpotrainer. --- ## Overview Direct Preference Optimization (DPO) in Unsloth provides a way to align models with human preferences using paired data (chosen/rejected). Unsloth optimizes this process by allowing `ref_model=None`, significantly reducing memory requirements while maintaining high performance. ## When to Use - When you have preference pairs (a better and worse response to the same prompt). - When RLHF is desired but VRAM is limited (preventing the loading of a second reference model). - When aligning reasoning or tone after a standard SFT phase. ## Decision Tree 1. Do you have a reference model already loaded? - No: Set `ref_model = None` in `DPOTrainer` to save VRAM. 2. Is your GPU RTX 40 series or H100? - Yes: Use FP8 preference optimization for faster training. 3. Is the model collapsing or losing its original abilities? - Yes: Lower the learning rate to 5e-6 and adjust the beta parameter (e.g., 0.1). ## Workflows ### DPO Trainer Initialization 1.
- Overview
- When to Use
- Decision Tree
- Workflows
- DPO Trainer Initialization
- Optimizing DPO VRAM Use
- Non-Obvious Insights
- Evidence
- Scripts
- Dependencies
- References
What does the unsloth-dpo skill do?
Direct Preference Optimization (DPO) in Unsloth provides a way to align models with human preferences using paired data (chosen/rejected). Unsloth optimizes this process by allowing refmodel=None, significantly reducing memory requirements while maintaining high performance.
How do I install it?
Run `npx skills add majiayu000/claude-skill-registry --skill unsloth-dpo-cuba6112-skillfactory-f41c8835 --agent claude-code` — it drops the skill into your project so the agent can pick it up. Swap the --agent value for codex, cursor or copilot if you use one of those.
Where does this skill come from?
From majiayu000/claude-skill-registry, a repository with 534 stars. We read it straight from the repository tree rather than a submitted listing, so what you see here is what is actually published.
Is a popular skill a good skill?
Not necessarily. Stars measure attention, not adoption — a repository can trend for a week and be abandoned. That is why we show the weekly change from our own snapshots next to the total, instead of a single flattering number.