pydantic-evals
Test and evaluate AI agents and LLM outputs using code-first evaluation framework with strong typing. Use when the user wants to: (1) Create evaluation datasets with test cases for AI agents, (2) Define evaluators (deterministic, LLM-as-Judge, custom, or span-based), (3) Run evaluations and generate reports, (4) Compare model performance across experiments, (5) Integrate evaluations with Pydantic AI agents, (6) Set up observability with Logfire, (7) Generate test datasets using LLMs, (8) Implement regression testing for AI systems.
npx skills add majiayu000/claude-skill-registry --skill pydantic-evals --agent claude-code
Same command for any agent — swap --agent for codex, cursor, copilot.
Weekly change comes from our own snapshots, not the repository page — it measures attention, not adoption.
# Pydantic Evals ## Overview Pydantic Evals provides rigorous testing and evaluation for AI agents and LLM outputs using a code-first approach with Pydantic models. It enables "Evaluation-Driven Development" (EDD) where evaluation suites live alongside application code, subject to version control and CI/CD. ## Core Concepts Understand these key primitives: ### Case A single test scenario with inputs, optional expected output, and metadata. ```python from pydantic_evals import Case case = Case( name="refund_request", inputs="What is your refund policy?", expected_output="30 days full refund", metadata={"category": "policy"} ) ``` ### Dataset Collection of Cases with default evaluators. Generic over input/output types. ```python from pydantic_evals import Dataset dataset = Dataset( cases=[case1, case2, case3], evaluators=[evaluator1, evaluator2] ) ``` ### Evaluator Logic engine that assesses outputs. Returns bool (Pass/Fail), float/int (score), or str (label). ### Experiment Point-in-time performance capture when Dataset runs against a Task. **For detailed explanations**, see [references/core-concepts.md](references/core-concepts.md) ## Quick Start Create and run a simple evaluation:
- Overview
- Core Concepts
- Case
- Dataset
- Evaluator
- Experiment
- Quick Start
- Evaluator Types
- 1. Deterministic Evaluators
- 2. LLM-as-a-Judge
- 3. Custom Evaluators
- 4. Span-Based Evaluation
- Integration with Pydantic AI
- Define Agent as Task
What does the pydantic-evals skill do?
Test and evaluate AI agents and LLM outputs using code-first evaluation framework with strong typing. Use when the user wants to: (1) Create evaluation datasets with test cases for AI agents, (2) Define evaluators (deterministic, LLM-as-Judge, custom, or span-based), (3) Run evaluations and generate reports, (4) Compare model performance across experiments, (5) Integrate evaluations with Pydantic AI agents, (6) Set up observability with Logfire, (7) Generate test datasets using LLMs, (8) Implement regression testing for AI systems.
How do I install it?
Run `npx skills add majiayu000/claude-skill-registry --skill pydantic-evals --agent claude-code` — it drops the skill into your project so the agent can pick it up. Swap the --agent value for codex, cursor or copilot if you use one of those.
Where does this skill come from?
From majiayu000/claude-skill-registry, a repository with 534 stars. We read it straight from the repository tree rather than a submitted listing, so what you see here is what is actually published.
Is a popular skill a good skill?
Not necessarily. Stars measure attention, not adoption — a repository can trend for a week and be abandoned. That is why we show the weekly change from our own snapshots next to the total, instead of a single flattering number.