benchmark-design
Design and calibrate a multi-Case capability Benchmark and establish a traceable Formal Baseline.
npx skills add Prism-Shadow/penguin-harness --skill benchmark-design --agent claude-code
Same command for any agent — swap --agent for codex, cursor, copilot.
Weekly change comes from our own snapshots, not the repository page — it measures attention, not adoption.
What it does
Instructs to build a multi-Case Benchmark for one Test Agent, calibrate with one Run per Case, and record the selected frozen Pilot as the Formal Baseline. It changes the Benchmark but does not run or score the Test Agent itself. It delegates evaluation to run_subagent and uses the agent-evaluation tool. It stops after the Baseline and does not perform optimization.
How it works
- Validates inputs: Test Agent, target capability, evaluation Runtime, and access before starting.
- Writes a Capability Contract outlining observable process, weaker behaviors to distinguish, and potential state improvements.
- Plans Case set and allocations; privately records intended behavior per Case and a plausible shortcut; writes and leak-checks the initial Benchmark.
- Executes one valid evaluation per planned Case to form Pilot iteration 1; completes this set before refining Cases.
- For later iterations, uses Scores and Traces to reconstruct solutions per Case; may refine multiple Cases per iteration.
- Freezes the first valid Pilot revision meeting the desired baseline score; if none, freezes the lowest-scoring valid Pilot revision.
- Freezes the selected revision and records the complete one-Run-per-Case Pilot result as the Formal Baseline when all cells are valid, agent state unchanges, and no design defects remain.
When to use it
- Triggered when you need to design and calibrate a formal capability Benchmark with a single Test Agent and a predefined baseline, requiring a formal baseline and controlled evaluation flow.
What it can touch
- Uses the environment to locate TEST_AGENT_DIR and BENCHMARK_DIR, SCOREBOARD, and relies on run_subagent with the agent-evaluation Skill. It requires explicit provider/model_id pairing and thinking_level in the evaluation results. The protocol for each run must be a YAML document with fields including provider, model_id, thinking_level, etc.
Caveats
- Aims to produce a Formal Baseline and stop before any optimization. It requires a valid Pilot iteration limit and properly resolved (provider, model_id). If inputs are missing, it asks for them before proceeding. It enforces that evaluation results must report the exact specified provider/model_id and thinking_level; mismatches invalidate the result and may require repair or retry.
- The Formal Baseline is recorded only when all criteria are met and no design defects remain; the score does not determine validity.
# Benchmark Design Build a multi-Case Benchmark for one Test Agent, calibrate its difficulty with one Run per Case, and record the selected frozen Pilot as the Formal Baseline. This Skill changes the Benchmark, never the Test Agent. It does not run or score the Test Agent. Delegate every evaluation with `run_subagent`, and tell each worker to use `agent-evaluation`. Stop after the Baseline; do not begin optimization. ## Before you start If the request does not identify a Test Agent, target capability, desired baseline score, and Pilot iteration limit, ask for the missing inputs. When they are already supplied, proceed without asking the user to restate them. Treat the current Agent as the **Builder**. A user-specified evaluation `(provider, model_id)` takes priority; otherwise inherit the current Builder Session's complete `Provider` and `Model ID` from the Environment. Never use a Project default as an implicit evaluation runtime. ## Workflow - A **Pilot** is a one-Run-per-Case evaluation used to improve the Benchmark. Unselected Pilot results never enter the Scoreboard; the selected result becomes the Formal Baseline after Freeze. - **Freeze** means the Benchmark revision and eva
- Before you start
- Workflow
- Setup and access
- Build the Benchmark
- Delegate evaluation
- Refine the Benchmark
- Freeze and record the Formal Baseline
- Record and finish
What does the benchmark-design skill do?
Design and calibrate a multi-Case capability Benchmark and establish a traceable Formal Baseline.
How do I install it?
Run `npx skills add Prism-Shadow/penguin-harness --skill benchmark-design --agent claude-code` — it drops the skill into your project so the agent can pick it up. Swap the --agent value for codex, cursor or copilot if you use one of those.
Where does this skill come from?
From Prism-Shadow/penguin-harness, a repository with 473 stars. We read it straight from the repository tree rather than a submitted listing, so what you see here is what is actually published.
Is a popular skill a good skill?
Not necessarily. Stars measure attention, not adoption — a repository can trend for a week and be abandoned. That is why we show the weekly change from our own snapshots next to the total, instead of a single flattering number.