Benchmark Manager
Create and manage AILANG eval benchmarks. Use when user asks to create benchmarks, fix benchmark issues, debug failing benchmarks, or analyze benchmark results.
npx skills add majiayu000/claude-skill-registry --skill benchmark-manager --agent claude-code
Same command for any agent — swap --agent for codex, cursor, copilot.
Weekly change comes from our own snapshots, not the repository page — it measures attention, not adoption.
# Benchmark Manager Manage AILANG evaluation benchmarks with correct prompt integration, debugging workflows, and best practices learned from real benchmark failures. ## Quick Start **Debugging a failing benchmark:** ```bash # 1. Show the full prompt that models see .claude/skills/benchmark-manager/scripts/show_full_prompt.sh json_parse # 2. Test a benchmark with a specific model ailang eval-suite --models claude-haiku-4-5 --benchmarks json_parse # 3. Check benchmark YAML for common issues .claude/skills/benchmark-manager/scripts/check_benchmark.sh benchmarks/json_parse.yml ``` ## When to Use This Skill Invoke this skill when: - User asks to create a new benchmark - User asks to debug/fix a failing benchmark - User wants to understand why models generate wrong code - User asks about benchmark YAML format - Benchmarks show 0% pass rate despite language support ## CRITICAL: prompt vs task_prompt **This is the most important concept for benchmark management.** ### The Problem (v0.4.8 Discovery) Benchmarks have TWO different prompt fields with VERY different behavior: | Field | Behavior | Use When | |-------|----------|----------| | `prompt:` | **REPLACES** the teaching prompt entirely
- Quick Start
- When to Use This Skill
- CRITICAL: prompt vs taskprompt
- The Problem (v0.4.8 Discovery)
- Why This Matters
- How Prompts Combine
- Available Scripts
- scripts/showfullprompt.sh
- scripts/checkbenchmark.sh
- scripts/testbenchmark.sh
- Benchmark YAML Format
- Required Fields
- Capability Names
- Creating New Benchmarks
ailang eval-suite --models claude-haiku-4-5 --benchmarks json_parse Check for issues Test with cheap model first ailang eval-suite --models claude-haiku-4-5 --benchmarks my_new_benchmark grep -E "^prompt:" benchmarks/failing_benchmark.yml If this returns a match, change to task_prompt: After editing prompts/v0.x.x.md, you MUST rebuild: make quick-install After editing prompts/v0.x.x.md: make quick-install # REQUIRED!
What does the Benchmark Manager skill do?
Create and manage AILANG eval benchmarks. Use when user asks to create benchmarks, fix benchmark issues, debug failing benchmarks, or analyze benchmark results.
How do I install it?
Run `npx skills add majiayu000/claude-skill-registry --skill benchmark-manager --agent claude-code` — it drops the skill into your project so the agent can pick it up. Swap the --agent value for codex, cursor or copilot if you use one of those.
Where does this skill come from?
From majiayu000/claude-skill-registry, a repository with 534 stars. We read it straight from the repository tree rather than a submitted listing, so what you see here is what is actually published.
Is a popular skill a good skill?
Not necessarily. Stars measure attention, not adoption — a repository can trend for a week and be abandoned. That is why we show the weekly change from our own snapshots next to the total, instead of a single flattering number.