ai-evals
Create an AI Evals Pack (eval PRD, test set, rubric, judge plan, results + iteration loop). Use for LLM evaluation, benchmarks, rubrics, error analysis/open coding, and ship/no-ship quality gates for AI features.
npx skills add majiayu000/claude-skill-registry --skill ai-evals --agent claude-code
Same command for any agent — swap --agent for codex, cursor, copilot.
Weekly change comes from our own snapshots, not the repository page — it measures attention, not adoption.
# AI Evals ## Scope **Covers** - Designing evaluation (“evals”) for LLM/AI features as an execution contract: what “good” means and how it’s measured - Converting failures into a **golden test set** + **error taxonomy** + **rubric** - Choosing a judging approach (human, LLM-as-judge, automated checks) and a repeatable harness/runbook - Producing decision-ready results and an iteration loop (every bug becomes a new test) **When to use** - “Design evals for this LLM feature so we can ship with confidence.” - “Create a rubric + golden set + benchmark for our AI assistant/copilot.” - “We’re seeing flaky quality—do error analysis and turn it into a repeatable eval.” - “Compare prompts/models safely with a clear acceptance threshold.” **When NOT to use** - You need to decide *what to build* (use `problem-definition`, `building-with-llms`, or `ai-product-strategy`). - You’re primarily doing traditional non-LLM software testing (use your standard eng QA/unit/integration tests). - You want model training research or infra design (this skill assumes API/model usage; delegate to ML/infra). - You only want vendor/model selection with no defined task + data (use `evaluating-new-technology` firs
- Scope
- Inputs
- Outputs (deliverables)
- Workflow (7 steps)
- 1) Define the decision and write the Eval PRD
- 2) Draft the golden set structure + coverage plan
- 3) Run error analysis and open coding to build a taxonomy
- 4) Convert taxonomy → rubric + scoring rules
- 5) Choose the judging approach + harness/runbook
- 6) Define reporting, thresholds, and the iteration loop
- 7) Quality gate + finalize
- Quality gate (required)
- Examples
What does the ai-evals skill do?
Create an AI Evals Pack (eval PRD, test set, rubric, judge plan, results + iteration loop). Use for LLM evaluation, benchmarks, rubrics, error analysis/open coding, and ship/no-ship quality gates for AI features.
How do I install it?
Run `npx skills add majiayu000/claude-skill-registry --skill ai-evals --agent claude-code` — it drops the skill into your project so the agent can pick it up. Swap the --agent value for codex, cursor or copilot if you use one of those.
Where does this skill come from?
From majiayu000/claude-skill-registry, a repository with 534 stars. We read it straight from the repository tree rather than a submitted listing, so what you see here is what is actually published.
Is a popular skill a good skill?
Not necessarily. Stars measure attention, not adoption — a repository can trend for a week and be abandoned. That is why we show the weekly change from our own snapshots next to the total, instead of a single flattering number.