preference-optimization
Align a fine-tuned model with preference data using DPO, ORPO, KTO, or SimPO. Use when preference pairs or thumbs-up/down feedback exist, when choosing between preference-optimization methods, or when a DPO run needs hyperparameters or debugging.
Profile →npx skills add wshobson/agents --skill preference-optimization --agent claude-code
Same command for any agent — swap --agent for codex, cursor, copilot.
Weekly change comes from our own snapshots, not the repository page — it measures attention, not adoption.
# Preference Optimization This skill assumes `finetuning-method-selection` already routed here because the data shape is preference pairs or unpaired thumbs-up/down feedback, not demonstrations (that's `lora-qlora-recipes`) or a verifiable reward signal (that's `grpo-rlvr-training`). What follows is method selection among the DPO family, the evidence for how much that selection actually matters, the production training pattern, and how to build the pairs in the first place. **Input:** a routing decision (preference optimization) plus preference pairs or unpaired feedback, usually from an SFT checkpoint. **Output format:** a validated method choice plus a config — the kwarg values in `references/method-configs.md`, not free-form advice — that `llm-finetuning-training-engineer` consumes directly. ## Method Selection | Data shape | Method | Key parameters | |---|---|---| | Preference pairs, default case | **DPO** | β=0.1, LR 5e-7–1e-6, 1–2 epochs | | Memory-bound or no SFT checkpoint | **ORPO** | reference-free, fused SFT+preference in one loss | | Unpaired thumbs-up/down | **KTO** | binary label per example, no pairing needed | | Length bias observed, sweep budget available | **SimPO
- Method Selection
- Worked Examples
- The Low-Leverage Truth
- Production Pattern: Iterative On-Policy DPO
- Pair Construction
- References
What does the preference-optimization skill do?
Align a fine-tuned model with preference data using DPO, ORPO, KTO, or SimPO. Use when preference pairs or thumbs-up/down feedback exist, when choosing between preference-optimization methods, or when a DPO run needs hyperparameters or debugging.
How do I install it?
Run `npx skills add wshobson/agents --skill preference-optimization --agent claude-code` — it drops the skill into your project so the agent can pick it up. Swap the --agent value for codex, cursor or copilot if you use one of those.
Where does this skill come from?
From wshobson/agents, a repository with 38,479 stars. We read it straight from the repository tree rather than a submitted listing, so what you see here is what is actually published.
Is a popular skill a good skill?
Not necessarily. Stars measure attention, not adoption — a repository can trend for a week and be abandoned. That is why we show the weekly change from our own snapshots next to the total, instead of a single flattering number.