dpo
Direct Preference Optimization for learning from preference pairs. Covers DPOTrainer, preference dataset preparation, implicit reward modeling, and beta tuning for stable preference learning without explicit reward models. Includes thinking quality patterns.
npx skills add majiayu000/claude-skill-registry --skill dpo --agent claude-code
Same command for any agent — swap --agent for codex, cursor, copilot.
Weekly change comes from our own snapshots, not the repository page — it measures attention, not adoption.
# Direct Preference Optimization (DPO) ## Overview DPO learns from preference pairs (chosen vs rejected responses) without training an explicit reward model. It directly optimizes the policy using the Bradley-Terry preference model, making it simpler than RLHF while achieving comparable results. This skill includes patterns for training thinking/reasoning models. ## Quick Reference | Component | Purpose | |-----------|---------| | `DPOTrainer` | Trainer for preference optimization | | `DPOConfig` | Training hyperparameters | | `beta` | Temperature for implicit reward (0.1 typical) | | `learning_rate` | 5e-6 (most conservative of RL methods) | | `ref_model` | Reference model for KL constraint | | Token ID 151668 | `</think>` boundary for Qwen3-Thinking models | ## Critical Environment Setup ```python import os from dotenv import load_dotenv load_dotenv() # Force text-based progress in Jupyter os.environ["TQDM_NOTEBOOK"] = "false" ``` ## Critical Import Order ```python # CRITICAL: Import unsloth FIRST for proper TRL patching import unsloth from unsloth import FastLanguageModel, is_bf16_supported # Then TRL imports from trl import DPOConfig, DPOTrainer from datasets import Dataset imp
- Overview
- Quick Reference
- Critical Environment Setup
- Critical Import Order
- DPO Concepts
- How DPO Works
- Key Differences from RLHF
- Dataset Format
- Required Fields
- From Comparison Data
- Thinking Quality Preference Pairs
- Setup
- Load Model
- Apply LoRA
What does the dpo skill do?
Direct Preference Optimization for learning from preference pairs. Covers DPOTrainer, preference dataset preparation, implicit reward modeling, and beta tuning for stable preference learning without explicit reward models. Includes thinking quality patterns.
How do I install it?
Run `npx skills add majiayu000/claude-skill-registry --skill dpo --agent claude-code` — it drops the skill into your project so the agent can pick it up. Swap the --agent value for codex, cursor or copilot if you use one of those.
Where does this skill come from?
From majiayu000/claude-skill-registry, a repository with 534 stars. We read it straight from the repository tree rather than a submitted listing, so what you see here is what is actually published.
Is a popular skill a good skill?
Not necessarily. Stars measure attention, not adoption — a repository can trend for a week and be abandoned. That is why we show the weekly change from our own snapshots next to the total, instead of a single flattering number.