agent-harness
Test and evaluation harness for AI agents — scenario suites, deterministic replay, regression diffing, cost and latency budgets. Use when agent quality is vibe-checked, before shipping a prompt or model change, or when evals drift.
npx skills add majiayu000/claude-skill-registry --skill agent-harness --agent claude-code
Same command for any agent — swap --agent for codex, cursor, copilot.
Weekly change comes from our own snapshots, not the repository page — it measures attention, not adoption.
# Agent Harness Most agents ship on vibes: someone tries eight prompts, the output looks good, it goes to production, and the next prompt tweak silently breaks a refusal nobody re-tested. This skill builds the harness around an agent so its behaviour becomes measurable — scenario suites with structural assertions, deterministic replay of recorded tool calls, paired regression diffing across prompt and model changes, and per-scenario cost and latency budgets. The tools here score an agent; they never invoke one, so they run offline on every commit. ## When to use this skill - An agent is going to production and the only quality evidence is manual spot-checking - A prompt, tool schema, or model version is changing and you need to know what broke - Two model or configuration options need a defensible comparison, not a demo - An incident happened and you need the behaviour encoded as a permanent regression test - Agent cost or latency is climbing across releases and nobody can point to when - An existing eval suite reports a healthy pass rate that nobody trusts ## Inputs the skill expects - The agent's tool inventory — names, arguments, and which tools are irreversible - Recorded trans
- When to use this skill
- Inputs the skill expects
- Clarify First
- Workflows
- Workflow 1 — Stand up a scenario suite and score a run
- Workflow 2 — Gate a prompt or model change on a paired regression diff
- Workflow 3 — Establish cost and latency budgets, then track drift
- Decision frameworks
- Which assertion type to reach for
- Severity, and what each one gates
- Can I trust this diff?
- Anti-Patterns
- Gating on the aggregate pass rate
- Asserting on sentences instead of structure
python3 engineering/agent-harness/scripts/scenario_runner.py \ python3 engineering/agent-harness/scripts/eval_diff.py \
What does the agent-harness skill do?
Test and evaluation harness for AI agents — scenario suites, deterministic replay, regression diffing, cost and latency budgets. Use when agent quality is vibe-checked, before shipping a prompt or model change, or when evals drift.
How do I install it?
Run `npx skills add majiayu000/claude-skill-registry --skill agent-harness --agent claude-code` — it drops the skill into your project so the agent can pick it up. Swap the --agent value for codex, cursor or copilot if you use one of those.
Where does this skill come from?
From majiayu000/claude-skill-registry, a repository with 534 stars. We read it straight from the repository tree rather than a submitted listing, so what you see here is what is actually published.
Is a popular skill a good skill?
Not necessarily. Stars measure attention, not adoption — a repository can trend for a week and be abandoned. That is why we show the weekly change from our own snapshots next to the total, instead of a single flattering number.