rag-eval
Use when the user has a RAG (Retrieval-Augmented Generation) system and wants to evaluate its quality — separating retrieval issues from generation issues. Also use when the user mentions RAG evaluation, faithfulness checking, hallucination detection in RAG, retrieval quality, chunking optimization, or "is my RAG pipeline working." Outputs a diagnostic matrix that pinpoints whether problems are in retrieval or generation.
Profile →npx skills add agentscope-ai/OpenJudge --skill 05-rag-eval --agent claude-code
Same command for any agent — swap --agent for codex, cursor, copilot.
Weekly change comes from our own snapshots, not the repository page — it measures attention, not adoption.
# RAG Eval Evaluate RAG systems by diagnosing retrieval and generation separately. A single "RAG accuracy" number hides whether the problem is finding the right documents or using them correctly. This skill separates them so you know what to fix. ## When to Activate - User has a RAG pipeline (retriever + generator) with traces - User wants to know if their RAG system hallucinates - User is optimizing chunking strategy and needs before/after comparison - User wants to build a RAG evaluation dataset ## Checklist You MUST create a task for each item and complete them in order: 1. **Load RAG traces** — validate query + context + answer triples 2. **Separate retrieval vs generation** — determine which layers to evaluate 3. **Run faithfulness evaluation** — is the answer grounded in retrieved docs? 4. **Run retrieval evaluation** — are the right documents retrieved? 5. **Build diagnostic matrix** — cross-tabulate to find root cause 6. **Output findings** — prioritized issues with concrete fixes ## Fast path: run the bundled script Once each trace has a faithfulness judgment (and ideally a retrieval signal), build the retrieval-vs-generation diagnostic matrix with the bundled, tested scri
- When to Activate
- Checklist
- Fast path: run the bundled script
- Step 1: Load RAG Traces
- Step 2: Separate Retrieval vs Generation
- Step 3: Faithfulness Evaluation
- Step 4: Retrieval Evaluation
- Chunking Optimization
- Step 5: Diagnostic Matrix
- Step 6: Output
- Common Mistakes
- Next Skills
python scripts/rag_diagnostic.py --traces traces.jsonl
What does the rag-eval skill do?
Use when the user has a RAG (Retrieval-Augmented Generation) system and wants to evaluate its quality — separating retrieval issues from generation issues. Also use when the user mentions RAG evaluation, faithfulness checking, hallucination detection in RAG, retrieval quality, chunking optimization, or "is my RAG pipeline working." Outputs a diagnostic matrix that pinpoints whether problems are in retrieval or generation.
How do I install it?
Run `npx skills add agentscope-ai/OpenJudge --skill 05-rag-eval --agent claude-code` — it drops the skill into your project so the agent can pick it up. Swap the --agent value for codex, cursor or copilot if you use one of those.
Where does this skill come from?
From agentscope-ai/OpenJudge, a repository with 764 stars. We read it straight from the repository tree rather than a submitted listing, so what you see here is what is actually published.
Is a popular skill a good skill?
Not necessarily. Stars measure attention, not adoption — a repository can trend for a week and be abandoned. That is why we show the weekly change from our own snapshots next to the total, instead of a single flattering number.