eval-frameworks
Traditional software metrics (accuracy, F1) fail to capture the quality of LLM outputs. Evaluation frameworks like Ragas and DeepEval use "LLM-as-a-judge" to quantify subjective qualities like faithfulness, relevance, and professionalism.
npx skills add majiayu000/claude-skill-registry --skill eval-frameworks-cuba6112-skillfactory --agent claude-code
Same command for any agent — swap --agent for codex, cursor, copilot.
Weekly change comes from our own snapshots, not the repository page — it measures attention, not adoption.
--- name: eval-frameworks description: Evaluation framework patterns for RAG and LLMs, including faithfulness metrics, synthetic dataset generation, and LLM-as-a-judge patterns. Triggers: ragas, deepeval, llm-eval, faithfulness, hallucination-check, synthetic-data. --- # Evaluation Frameworks ## Overview Traditional software metrics (accuracy, F1) fail to capture the quality of LLM outputs. Evaluation frameworks like Ragas and DeepEval use "LLM-as-a-judge" to quantify subjective qualities like faithfulness, relevance, and professionalism. ## When to Use - **RAG Benchmarking**: To verify if answers are supported by retrieved context (Faithfulness). - **Regression Testing**: Ensuring that a prompt change or model upgrade doesn't break existing behavior. - **Synthetic Benchmarking**: Creating evaluation sets when manual gold-standard data is unavailable. ## Decision Tree 1. Do you want to check for hallucinations? - YES: Run a Faithfulness metric. 2. Is the retrieved context actually useful for the question? - YES: Run a Retrieval Relevance metric. 3. Do you need to scale evaluation without manual labeling? - YES: Use Synthetic Data Generation. ## Workflows ### 1. Evaluating RAG Faith
- Overview
- When to Use
- Decision Tree
- Workflows
- 1. Evaluating RAG Faithfulness
- 2. Unit Testing LLM Outputs (DeepEval)
- 3. Automated Question Generation
- Non-Obvious Insights
- Evidence
- Scripts
- Dependencies
- References
What does the eval-frameworks skill do?
Traditional software metrics (accuracy, F1) fail to capture the quality of LLM outputs. Evaluation frameworks like Ragas and DeepEval use "LLM-as-a-judge" to quantify subjective qualities like faithfulness, relevance, and professionalism.
How do I install it?
Run `npx skills add majiayu000/claude-skill-registry --skill eval-frameworks-cuba6112-skillfactory --agent claude-code` — it drops the skill into your project so the agent can pick it up. Swap the --agent value for codex, cursor or copilot if you use one of those.
Where does this skill come from?
From majiayu000/claude-skill-registry, a repository with 534 stars. We read it straight from the repository tree rather than a submitted listing, so what you see here is what is actually published.
Is a popular skill a good skill?
Not necessarily. Stars measure attention, not adoption — a repository can trend for a week and be abandoned. That is why we show the weekly change from our own snapshots next to the total, instead of a single flattering number.