agent-evals
Design and implement evaluation frameworks for AI agents. Use when testing agent reasoning quality, building graders, doing error analysis, or establishing regression protection. Framework-agnostic concepts that apply to any SDK.
npx skills add majiayu000/claude-skill-registry --skill agent-evals-panaversity-agentfactory-2 --agent claude-code
Same command for any agent — swap --agent for codex, cursor, copilot.
Weekly change comes from our own snapshots, not the repository page — it measures attention, not adoption.
# Agent Evaluations: Measuring Reasoning Quality **Core Thesis**: "One of the biggest predictors for whether someone is able to build agentic workflows really well is whether or not they're able to drive a really disciplined evaluation process." — Andrew Ng Evaluations (evals) are exams for your agent's reasoning. Unlike traditional testing (TDD) that checks code correctness with PASS/FAIL outcomes, evals measure reasoning quality with probabilistic scores. The distinction is critical: | Aspect | TDD (Code Testing) | Evals (Agent Evaluation) | |--------|-------------------|-------------------------| | **Tests** | Does function return correct output? | Did agent make the right decision? | | **Outcome** | PASS or FAIL (deterministic) | Scores (probabilistic) | | **Example** | "Does get_weather() return valid JSON?" | "Did agent correctly interpret user intent?" | | **Analogy** | Testing if calculator works | Testing if student knows WHEN to use multiplication | ## When to Activate Activate this skill when: - Building systematic quality checks for any AI agent - Designing evaluation datasets (typical, edge, error cases) - Creating graders to define "good" automatically - Performing er
- When to Activate
- Core Concepts
- 1. Evals as Exams
- 2. The Two Evaluation Axes
- 3. Graders
- 4. Error Analysis (Most Critical Skill)
- 5. End-to-End vs Component-Level Evals
- 6. Dataset Design
- 7. Regression Protection
- Practical Guidance
- Building Quick-and-Dirty Evals
- Creating Effective Graders
- LLM Grader Template
- Error Analysis Workflow
What does the agent-evals skill do?
Design and implement evaluation frameworks for AI agents. Use when testing agent reasoning quality, building graders, doing error analysis, or establishing regression protection. Framework-agnostic concepts that apply to any SDK.
How do I install it?
Run `npx skills add majiayu000/claude-skill-registry --skill agent-evals-panaversity-agentfactory-2 --agent claude-code` — it drops the skill into your project so the agent can pick it up. Swap the --agent value for codex, cursor or copilot if you use one of those.
Where does this skill come from?
From majiayu000/claude-skill-registry, a repository with 534 stars. We read it straight from the repository tree rather than a submitted listing, so what you see here is what is actually published.
Is a popular skill a good skill?
Not necessarily. Stars measure attention, not adoption — a repository can trend for a week and be abandoned. That is why we show the weekly change from our own snapshots next to the total, instead of a single flattering number.