ai-evals
Help users build robust infrastructure for measuring, monitoring, and iterating on AI product performance using human, code-based, and LLM-as-a-judge methodologies.
npx skills add RefoundAI/lenny-skills --skill ai-evals --agent claude-code
Same command for any agent — swap --agent for codex, cursor, copilot.
Weekly change comes from our own snapshots, not the repository page — it measures attention, not adoption.
# AI Evaluation Strategy Move beyond vibe checks to systematic, empirical measurement of AI product quality and reliability. Help the user with ai evaluation strategy using insights from 11 guests and posts across Lenny's Podcast and Newsletter. ## How to Help 1. **Identify Failure Modes** - Help the user conduct error analysis on real traces to find where the system specifically breaks. 2. **Select Eval Methods** - Recommend the right mix of human, code, and LLM judges based on the specific technical use case. 3. **Build Gold Sets** - Assist in curating a reference dataset of high-quality examples to act as the ground truth for your application. 4. **Operationalize** - Guide the user in integrating these evaluations into a CI/CD pipeline for continuous quality improvement. ## Core Principles ### Automate the Value Chain Brendan Foody: "I think that for enterprises especially, the core way to think about it is how can they build a test or systematic way to measure how well AI automates their core value chain? So if it's an architecture firm that's producing these architecture diagrams of what they provide to their end customer, how can they effectively measure that? And each compan
- How to Help
- Core Principles
- Automate the Value Chain
- Prioritize Subjective Excellence
- Eliminate Vibe Checks
- Structured Judge Logic
- Templates & Frameworks
- Questions to Help Users
- Common Mistakes to Flag
- Deep Dive
- Related Skills
What does the ai-evals skill do?
Help users build robust infrastructure for measuring, monitoring, and iterating on AI product performance using human, code-based, and LLM-as-a-judge methodologies.
How do I install it?
Run `npx skills add RefoundAI/lenny-skills --skill ai-evals --agent claude-code` — it drops the skill into your project so the agent can pick it up. Swap the --agent value for codex, cursor or copilot if you use one of those.
Where does this skill come from?
From RefoundAI/lenny-skills, a repository with 1,215 stars. We read it straight from the repository tree rather than a submitted listing, so what you see here is what is actually published.
Is a popular skill a good skill?
Not necessarily. Stars measure attention, not adoption — a repository can trend for a week and be abandoned. That is why we show the weekly change from our own snapshots next to the total, instead of a single flattering number.