ai-evaluation-suite
Comprehensive AI/LLM evaluation toolkit for production AI systems. Covers LLM output quality, prompt engineering, RAG evaluation, agent performance, hallucination detection, bias assessment, cost/token optimization, latency metrics, model comparison, and fine-tuning evaluation. Includes BLEU/ROUGE metrics, perplexity, F1 scores, LLM-as-judge patterns, and benchmarks like MMLU and HumanEval.
npx skills add majiayu000/claude-skill-registry --skill ai-evaluation-suite --agent claude-code
Same command for any agent — swap --agent for codex, cursor, copilot.
Weekly change comes from our own snapshots, not the repository page — it measures attention, not adoption.
# AI Evaluation Suite - Quick Reference ## Purpose Production AI systems require rigorous evaluation beyond traditional software testing. This skill provides comprehensive evaluation capabilities for LLM quality, RAG systems, agents, hallucination detection, bias assessment, cost optimization, and performance metrics. ## When to Use This Skill - Evaluating LLM outputs for quality and correctness - A/B testing prompt variations - Measuring RAG system retrieval accuracy - Detecting hallucinations in generated content - Assessing model bias and fairness - Optimizing token usage and costs - Comparing multiple LLM models - Evaluating fine-tuned models vs base models - Running standard benchmarks (MMLU, HumanEval, etc.) - Implementing LLM-as-judge evaluation patterns ## Core Concepts ### Evaluation Pyramid ``` ┌─────────────┐ │ Human Eval │ <- Gold standard but expensive └──────┬──────┘ │ ┌──────┴──────────┐ │ LLM-as-Judge │ <- Scalable proxy for human judgment └──────┬──────────┘ │ ┌──────┴──────────┐ │ Reference-Based │ <- BLEU, ROUGE, F1 (needs ground truth) └──────┬──────────┘ │ ┌──────┴──────────┐ │ Reference-Free │ <- Perplexity, consistency, coherence └─────────────────┘ ``` ### K
- Purpose
- When to Use This Skill
- Core Concepts
- Evaluation Pyramid
- Key Metric Categories
- Documentation Structure
- Quick Start: LLM Quality Evaluation
- Quick Start: Hallucination Detection
- Quick Start: RAG Evaluation
- Quick Start: Prompt Engineering A/B Test
- Quick Start: Cost Optimization
- Best Practices
- DO's
- DON'Ts
What does the ai-evaluation-suite skill do?
Comprehensive AI/LLM evaluation toolkit for production AI systems. Covers LLM output quality, prompt engineering, RAG evaluation, agent performance, hallucination detection, bias assessment, cost/token optimization, latency metrics, model comparison, and fine-tuning evaluation. Includes BLEU/ROUGE metrics, perplexity, F1 scores, LLM-as-judge patterns, and benchmarks like MMLU and HumanEval.
How do I install it?
Run `npx skills add majiayu000/claude-skill-registry --skill ai-evaluation-suite --agent claude-code` — it drops the skill into your project so the agent can pick it up. Swap the --agent value for codex, cursor or copilot if you use one of those.
Where does this skill come from?
From majiayu000/claude-skill-registry, a repository with 534 stars. We read it straight from the repository tree rather than a submitted listing, so what you see here is what is actually published.
Is a popular skill a good skill?
Not necessarily. Stars measure attention, not adoption — a repository can trend for a week and be abandoned. That is why we show the weekly change from our own snapshots next to the total, instead of a single flattering number.