RAG Evaluation
Comprehensive guide to evaluating Retrieval-Augmented Generation systems including retrieval metrics, generation quality, faithfulness, and end-to-end evaluation frameworks
npx skills add majiayu000/claude-skill-registry --skill rag-evaluation --agent claude-code
Same command for any agent — swap --agent for codex, cursor, copilot.
Weekly change comes from our own snapshots, not the repository page — it measures attention, not adoption.
What it does
Describes how to evaluate RAG systems by measuring retrieval quality, context sufficiency, generation faithfulness and relevance, and end-to-end user outcomes. Presents multiple metrics (Precision@k, Recall@k, MRR, NDCG, context relevance/precision/recall, faithfulness, answer relevance, correctness, completeness), and outlines how to apply them across component, end-to-end, and user levels. Includes example formulas, prompts, and code sketches for faithfulness, relevance, and context checks, plus references to evaluation frameworks (RAGAS, TruLens, DeepEval) and dataset creation structures.
How it works
The skill instructs an agent to perform systematic evaluation across three axes:
- Retrieval: compute metrics like Precision@k, Recall@k, MRR, NDCG to assess retrieved documents.
- Context: assess whether the retrieved context sufficiently supports the answer via relevance-based metrics.
- Generation: evaluate faithfulness (grounded in context), answer relevance, correctness, completeness, and safety using LLM-based judgments, NLI checks, and cosine similarity for semantic alignment. It provides concrete evaluation methods:
- Faithfulness Evaluation (LLM-as-Judge and NLI-based checks) with prompts and code skeletons.
- Answer Relevance Evaluation (LLM-as-Judge prompts and semantic similarity via embeddings).
- Correctness Evaluation (Exact match and F1 score against ground truth; LLM rubric prompts).
- RAG-specific metrics (Context Precision/Recall/Relevance) with definitions and example calculations.
- Dataset creation structures (Question-Answer pairs, Question-Context-Answer Triples).
- Framework integrations (RAGAS, TruLens, DeepEval, Langfuse) with usage snippets. Concrete actions include prompting, ranking, scoring, and aggregating results across the described metrics, plus optional tooling for dataset creation and evaluation pipelines.
When to use it
Use when evaluating or benchmarking a Retrieval-Augmented Generation system to quantify retrieval quality, contextual sufficiency, and answer faithfulness and relevance, either for component-level analysis, end-to-end assessment, or user-satisfaction studies. It is applicable when constructing evaluation datasets and when integrating with frameworks like ragas, TruLens, or DeepEval.
What it can touch
No explicit tool execution is listed beyond code snippets and prompts. Tools mentioned for evaluation workflows include:
- ragas (Python package) for evaluation and metrics
- TruLens for real-time evaluation and dashboards
- DeepEval for multiple metrics and LLM-as-judge
- Langfuse for observability
- Custom evaluation scripts as shown in code blocks
Caveats
License indicated as MIT. The material provides formulas, prompts, and code skeletons but does not guarantee outcomes; results depend on the underlying models and data. No limitations beyond what's described in the sections and examples; no additional caveats are stated.
# RAG Evaluation ## What is RAG Evaluation? **Definition:** Measuring quality of both retrieval and generation components in RAG systems. ### Components ``` RAG System = Retrieval + Generation Retrieval: Query → Relevant documents Generation: Documents + Query → Answer Both need evaluation! ``` ### Evaluation Levels ``` 1. Component-level: Retrieval quality, generation quality 2. End-to-end: Overall answer quality 3. User-level: User satisfaction, task success ``` --- ## Why RAG Evaluation Matters ### RAG Quality Varies Widely **Example:** ``` Query: "What is the capital of France?" Bad RAG: - Retrieves: Document about French cuisine - Generates: "France is known for its wine" (irrelevant) Good RAG: - Retrieves: Document about Paris - Generates: "The capital of France is Paris" (correct) ``` ### Retrieval Errors → Wrong Context → Bad Answers **Error Cascade:** ``` Poor retrieval (irrelevant docs) → Wrong context for LLM → LLM generates answer from wrong info → Hallucination or incorrect answer ``` ### Need Metrics to Improve Systematically **Without Metrics:** ``` "This answer seems wrong" → Unclear what to fix ``` **With Metrics:** ``` Context Precision: 0.3 (low) → Improve retrie
- What is RAG Evaluation?
- Components
- Evaluation Levels
- Why RAG Evaluation Matters
- RAG Quality Varies Widely
- Retrieval Errors → Wrong Context → Bad Answers
- Need Metrics to Improve Systematically
- RAG Components to Evaluate
- 1. Retrieval: Are Relevant Docs Retrieved?
- 2. Context: Is Context Sufficient for Answer?
- 3. Generation: Is Answer Correct, Relevant, Safe?
- Retrieval Evaluation Metrics
- Precision@k: % of Top-k Results Relevant
- Recall@k: % of Relevant Docs in Top-k
pip install ragas
What does the RAG Evaluation skill do?
Comprehensive guide to evaluating Retrieval-Augmented Generation systems including retrieval metrics, generation quality, faithfulness, and end-to-end evaluation frameworks
How do I install it?
Run `npx skills add majiayu000/claude-skill-registry --skill rag-evaluation --agent claude-code` — it drops the skill into your project so the agent can pick it up. Swap the --agent value for codex, cursor or copilot if you use one of those.
Where does this skill come from?
From majiayu000/claude-skill-registry, a repository with 534 stars. We read it straight from the repository tree rather than a submitted listing, so what you see here is what is actually published.
Is a popular skill a good skill?
Not necessarily. Stars measure attention, not adoption — a repository can trend for a week and be abandoned. That is why we show the weekly change from our own snapshots next to the total, instead of a single flattering number.