agent-benchmark
Framework for measuring and tracking agent response quality over time. Detects regressions before they reach production. Use when evaluating agent changes, auditing quality, or establishing performance baselines.
npx skills add vibeeval/vibecosystem --skill agent-benchmark --agent claude-code
Same command for any agent — swap --agent for codex, cursor, copilot.
Weekly change comes from our own snapshots, not the repository page — it measures attention, not adoption.
# Agent Benchmark Framework Without benchmarks, we cannot know whether agent changes improve or degrade quality. This skill defines how to measure, track, and protect agent performance. ## When to Activate - Before and after modifying any agent definition file - When adding a new skill that an agent depends on - Periodic quality audits (weekly/monthly) - When a user reports degraded agent output - Before promoting an agent from experimental to production ## Core Concepts ### Why Benchmarks Matter Agent quality degrades silently. A prompt tweak that improves one response can break ten others. Without a baseline to compare against, every change is a guess. Benchmarks make quality visible and regressions detectable. ### Benchmark Types | Type | Scope | Cost | Frequency | |------|-------|------|-----------| | Prompt Benchmark | Single agent, single task | Low | Every agent change | | Task Benchmark | End-to-end scenario | Medium | Feature changes | | Regression Suite | All critical agents | High | Weekly / before release | ## Directory Structure ``` ~/.claude/benchmarks/ fixtures/ code-reviewer/ missing-error-handling.ts # Input: code with no try/catch sql-injection.py # Input: unparam
- When to Activate
- Core Concepts
- Why Benchmarks Matter
- Benchmark Types
- Directory Structure
- Scoring Rubric Template
- Ground Truth Format
- Scoring Logic
- How a Run Is Scored
- Score Interpretation
- Running Benchmarks
- Run All Benchmarks
- Run Single Agent
- Compare Against Baseline
Full suite node ~/.claude/benchmarks/run.mjs Benchmark one agent node ~/.claude/benchmarks/run.mjs --agent code-reviewer With verbose output (shows actual vs expected per fixture) node ~/.claude/benchmarks/run.mjs --agent code-reviewer --verbose Compare latest run against saved baseline node ~/.claude/benchmarks/run.mjs --compare Compare specific run against specific baseline node ~/.claude/benchmarks/run.mjs \
What does the agent-benchmark skill do?
Framework for measuring and tracking agent response quality over time. Detects regressions before they reach production. Use when evaluating agent changes, auditing quality, or establishing performance baselines.
How do I install it?
Run `npx skills add vibeeval/vibecosystem --skill agent-benchmark --agent claude-code` — it drops the skill into your project so the agent can pick it up. Swap the --agent value for codex, cursor or copilot if you use one of those.
Where does this skill come from?
From vibeeval/vibecosystem, a repository with 521 stars. We read it straight from the repository tree rather than a submitted listing, so what you see here is what is actually published.
Is a popular skill a good skill?
Not necessarily. Stars measure attention, not adoption — a repository can trend for a week and be abandoned. That is why we show the weekly change from our own snapshots next to the total, instead of a single flattering number.