eval-harness-first
Build the evaluation harness that gates every fine-tuning run — golden sets, per-failure-mode graders, judge calibration, and base-model baselines. Use when starting a fine-tuning effort, when converting traces into an eval set, or when calibrating a judge against human labels.
Profile →npx skills add wshobson/agents --skill eval-harness-first --agent claude-code
Same command for any agent — swap --agent for codex, cursor, copilot.
Weekly change comes from our own snapshots, not the repository page — it measures attention, not adoption.
# Eval Harness First The Phase 0 gate for the whole plugin: `finetuning-method-selection` and every downstream skill assume this harness exists before a training config gets written. The harness is not a run-end side artifact — it is the data-curation engine. The same labeled traces that build the goldens feed training data, minus an explicit holdout. **Input:** production/agent traces if they exist, or a task spec if they don't, plus labelers willing to grade ≥100 examples. **Output format:** the `eval/` directory below — goldens, graders, drift suite, and the base-model baseline that later phases gate on. ## The Gate No eval harness, no fine-tune. Skip to a training config and there is nothing to measure against, nothing to catch regressions, and no labeled data to train on. The flywheel: 1. **Collect traces** — production/agent spans, or synthetic tasks if none exist yet. 2. **Error analysis** — open coding on ≥100 traces, axial coding into 4–8 failure buckets. 3. **One grader per bucket** — deterministic first; calibrated LLM-judge only for genuinely subjective criteria. 4. **Prioritize** by frequency × severity × value. 5. **The labeled traces feed dataset curation, minus an e
- The Gate
- Building Goldens
- Graders
- Judge Calibration Is a Prerequisite
- The Baseline
- Directory Contract
- Phase 0 Exit Checklist
- Related Skills
- References
What does the eval-harness-first skill do?
Build the evaluation harness that gates every fine-tuning run — golden sets, per-failure-mode graders, judge calibration, and base-model baselines. Use when starting a fine-tuning effort, when converting traces into an eval set, or when calibrating a judge against human labels.
How do I install it?
Run `npx skills add wshobson/agents --skill eval-harness-first --agent claude-code` — it drops the skill into your project so the agent can pick it up. Swap the --agent value for codex, cursor or copilot if you use one of those.
Where does this skill come from?
From wshobson/agents, a repository with 38,479 stars. We read it straight from the repository tree rather than a submitted listing, so what you see here is what is actually published.
Is a popular skill a good skill?
Not necessarily. Stars measure attention, not adoption — a repository can trend for a week and be abandoned. That is why we show the weekly change from our own snapshots next to the total, instead of a single flattering number.