trace-to-training-data
Convert evaluation traces and production logs into SFT examples and preference pairs. Use when graded traces or failure examples exist and need to become training data, when applying rejection sampling to model outputs, or when building DPO pairs from passing and failing runs.
Profile →npx skills add wshobson/agents --skill trace-to-training-data --agent claude-code
Same command for any agent — swap --agent for codex, cursor, copilot.
Weekly change comes from our own snapshots, not the repository page — it measures attention, not adoption.
# Trace To Training Data This skill assumes `eval-harness-first` already graded the traces being converted here — goldens, graders, and `runs/<run-id>/results.json` all exist before conversion starts. This is the flywheel edge that skill names in its own flow: "the same labeled traces become the training set." Conversion happens here; grading already happened upstream. **Input:** graded traces — `eval/goldens.jsonl` plus `runs/<run-id>/results.json`, each row carrying a `task_id`, a `verdict` from the grader, and a `reward` when the task supports a scalar score (judge score, execution partial-credit, or an RLVR verifier): ```json {"task_id": "t-042", "trace_id": "t-042-a3", "messages": [{"role": "user", "content": "..."}], "verdict": "pass", "reward": 0.91, "grader": "exact_match"} ``` **Output format:** rows shaped exactly like `dataset-curation`'s Format Selection table — SFT `messages` rows or DPO `prompt`/`chosen`/`rejected` pairs — so this skill's output is that skill's input with no reshaping step in between. ## The Principle The eval harness already did the labeling work: every trace in `results.json` carries a verdict, and often a reward, before this skill ever touches it.
- The Principle
- SFT From Traces
- Preference Pairs From Traces
- Hygiene
- Related Skills
What does the trace-to-training-data skill do?
Convert evaluation traces and production logs into SFT examples and preference pairs. Use when graded traces or failure examples exist and need to become training data, when applying rejection sampling to model outputs, or when building DPO pairs from passing and failing runs.
How do I install it?
Run `npx skills add wshobson/agents --skill trace-to-training-data --agent claude-code` — it drops the skill into your project so the agent can pick it up. Swap the --agent value for codex, cursor or copilot if you use one of those.
Where does this skill come from?
From wshobson/agents, a repository with 38,479 stars. We read it straight from the repository tree rather than a submitted listing, so what you see here is what is actually published.
Is a popular skill a good skill?
Not necessarily. Stars measure attention, not adoption — a repository can trend for a week and be abandoned. That is why we show the weekly change from our own snapshots next to the total, instead of a single flattering number.