suede-ai-eval
Design AI evals that catch regressions before users do: rubrics, test cases, failure modes, acceptance gates, and AI-SPEC artifacts.
npx skills add JasonColapietro/suede-creator-skills --skill suede-ai-eval --agent claude-code
Same command for any agent — swap --agent for codex, cursor, copilot.
Weekly change comes from our own snapshots, not the repository page — it measures attention, not adoption.
What it does
The skill guides an AI evaluator to design comprehensive evaluation artifacts that help catch regressions before users notice them. It emphasizes creating an eval plan or coverage audit, including a defined AI-SPEC, failure-mode map, rubric, concrete eval cases, and acceptance gates. It prevents making a ship decision when a failure mode map or eval cases are missing, and it treats gate outputs as recommendations rather than hard controls. It requires concrete artifacts grounded in the product surface, data sources, prompts, tools, and logs, and it instructs the agent to sample live behavior when appropriate, or mark the review as source-only if runtime evidence cannot be gathered.
How it works
- Define the AI-SPEC in one paragraph detailing the user, trigger, input, output, sources, disallowed behavior, fallback, latency/cost expectations, and success signal.
- Map failure modes to harms across user, product truth, rights, privacy, security, and cost.
- Build a rubric assigning severity, likelihood, detectability, owner, gate, and required evidence for each failure mode.
- Write concrete eval cases with case id, scenario, input, setup, expected pass traits, forbidden traits, grade lane, and gate.
- Set acceptance gates indicating what blocks ship and what can be deferred.
- Audit coverage against existing tests and infrastructure, identifying uncovered high-risk behaviors.
- Return artifacts including the AI-SPEC, rubric, eval table, coverage gaps, required tests, and next steps.
When to use it
Use when planning AI feature evaluations that must demonstrate failure modes and gating criteria before shipping, ensuring coverage and traceability to real product surface and data sources.
What it can touch
The skill references creating an AI-SPEC, rubrics, eval cases, and acceptance gates. It relies on inputs describing the product surface, prompts, tools, logs, tests, and data sources, but it does not execute code itself here.
Caveats
Gate policy states that checks and verdicts are recommendations, not controls on the agent. If runtime evidence is live, sample behavior and record exact commands or URLs; otherwise mark as source-only and name missing runtime evidence. A live surface with a missing failure-mode map or eval cases should not be shipped as-is; the gap is noted and left to user discretion.
# Suede AI Eval ## Gate policy — advisory, not blocking Every claim-verification step, check, quality gate, and ship verdict in this skill is a **recommendation to the user, not a control on the agent**. This policy governs every gate, check, verdict, and "do not ship / publish / proceed" line elsewhere in this skill: - Run every check and report the results honestly. Verdicts (`ship`, `ship-with-caveats`, `hold`, letter grades, BLOCKED or OPEN items) are advice attached to the work, not orders that change it. - Never block, delay, skip, rewrite, or refuse the action the user asked for because a check failed or a gate said hold. Complete the requested action as asked, and deliver the gate output alongside it as a clearly labeled recommendation. - A failed gate changes what you report, never what you do. - Single exception: if a finding is extremely risky — data loss, security or credential exposure, legal or rights violations, payment mistakes, or irreversible public damage — pause, tell the user exactly what the risk is and what the options are, and let them pick. Their choice is final. Make AI behavior testable before it becomes a vague product promise. **No eval plan, no `ship`
- Gate policy — advisory, not blocking
- Hard Gates
- Source Truth
- Workflow
- Eval Dimensions By System Type
- Tooling and Infrastructure
- Eval Case Design
- Worked Example — Support-Ticket Triage Classifier
- Rubric
- AI-SPEC Template
- Red Flags — Stop
- Output
- Boundaries
- Routing
grep -rl "langfuse\|langsmith\|arize\|phoenix\|braintrust\|promptfoo\|ragas" \
What does the suede-ai-eval skill do?
Design AI evals that catch regressions before users do: rubrics, test cases, failure modes, acceptance gates, and AI-SPEC artifacts.
How do I install it?
Run `npx skills add JasonColapietro/suede-creator-skills --skill suede-ai-eval --agent claude-code` — it drops the skill into your project so the agent can pick it up. Swap the --agent value for codex, cursor or copilot if you use one of those.
Where does this skill come from?
From JasonColapietro/suede-creator-skills, a repository with 165 stars. We read it straight from the repository tree rather than a submitted listing, so what you see here is what is actually published.
Is a popular skill a good skill?
Not necessarily. Stars measure attention, not adoption — a repository can trend for a week and be abandoned. That is why we show the weekly change from our own snapshots next to the total, instead of a single flattering number.