evaluating-llms
Evaluate LLM systems using automated metrics, LLM-as-judge, and benchmarks. Use when testing prompt quality, validating RAG pipelines, measuring safety (hallucinations, bias), or comparing models for production deployment.
npx skills add majiayu000/claude-skill-registry --skill evaluating-llms --agent claude-code
Same command for any agent — swap --agent for codex, cursor, copilot.
Weekly change comes from our own snapshots, not the repository page — it measures attention, not adoption.
What it does
Evaluate Large Language Model (LLM) systems using automated metrics, LLM-as-judge patterns, and standardized benchmarks to ensure production quality and safety.
How it works
- It outlines when to use the skill for tasks such as testing prompts, validating RAG pipelines, measuring hallucinations or bias, comparing models, and running benchmark tests.
- It presents a decision framework for evaluation by task type (Classification, Generation, Question Answering, RAG Systems, Code Generation, Multi-step Agents) and by volume/cost, recommending a layered approach (Automated metrics, LLM-as-judge, Human evaluation).
- For each task type, it specifies evaluation methods (e.g., Exact Match, JSON Schema Validation, LLM-as-Judge) and lists relevant metrics and tools (e.g., BLEU, ROUGE, BERTScore; GPT-4, Claude; RAGAS; pytest; HumanEval).
- It provides quick-start code snippets and example workflows for unit evaluation, RAG evaluation, LLM-as-judge evaluation, safety/bias assessment, and benchmark usage.
- It includes quick reference tables and links to detailed references and working examples.
When to use it
Use this skill when testing prompts, validating RAG pipelines, measuring hallucinations, bias, or toxicity, comparing models or prompt configurations, running benchmarks (MMLU, HumanEval), and integrating quality checks into CI/CD or production monitoring. Common triggers include prompts about testing RAG systems, measuring hallucinations, evaluating generation quality, comparing GPT-4 vs Claude, and detecting bias.
What it can touch
- Tools: LLMs for judging and evaluation (GPT-4, Claude), RAGAS framework, libraries like lm-eval, scikit-learn, pytest, OpenAI Python client.
- It references code examples and datasets in Python and shell commands for benchmarking (e.g., lm_eval, RAGAS) and shows how to run Quick Start scripts.
Caveats
- The skill discusses multiple evaluation methods with probabilistic or heuristic metrics (e.g., LLM-as-Judge correlation with human judgment, automated metrics like BLEU/ROUGE whose limitations are noted).
- It emphasizes layered evaluation but does not guarantee production readiness or human agreement; it suggests best practices and potential limitations (e.g., automated metrics may not fully reflect subjective quality).
# LLM Evaluation Evaluate Large Language Model (LLM) systems using automated metrics, LLM-as-judge patterns, and standardized benchmarks to ensure production quality and safety. ## When to Use This Skill Apply this skill when: - Testing individual prompts for correctness and formatting - Validating RAG (Retrieval-Augmented Generation) pipeline quality - Measuring hallucinations, bias, or toxicity in LLM outputs - Comparing different models or prompt configurations (A/B testing) - Running benchmark tests (MMLU, HumanEval) to assess model capabilities - Setting up production monitoring for LLM applications - Integrating LLM quality checks into CI/CD pipelines Common triggers: - "How do I test if my RAG system is working correctly?" - "How can I measure hallucinations in LLM outputs?" - "What metrics should I use to evaluate generation quality?" - "How do I compare GPT-4 vs Claude for my use case?" - "How do I detect bias in LLM responses?" ## Evaluation Strategy Selection ### Decision Framework: Which Evaluation Approach? **By Task Type:** | Task Type | Primary Approach | Metrics | Tools | |-----------|------------------|---------|-------| | **Classification** (sentiment, intent) | A
- When to Use This Skill
- Evaluation Strategy Selection
- Decision Framework: Which Evaluation Approach?
- Core Evaluation Patterns
- Unit Evaluation (Individual Prompts)
- RAG (Retrieval-Augmented Generation) Evaluation
- LLM-as-Judge Evaluation
- Safety and Alignment Evaluation
- Benchmark Testing
- Production Evaluation
- Classification Task Evaluation
- Generation Task Evaluation
- Quick Reference Tables
- Evaluation Framework Selection
pip install lm-eval Evaluate GPT-4 on MMLU lm_eval --model openai-chat --model_args model=gpt-4 --tasks mmlu --num_fewshot 5 Run RAGAS evaluation on custom dataset python scripts/run_ragas_eval.py --dataset data/qa_dataset.json --output results.json Compare GPT-4 vs Claude on benchmark python scripts/compare_models.py --model-a gpt-4 --model-b claude-3-opus --tasks mmlu,humaneval
What does the evaluating-llms skill do?
Evaluate LLM systems using automated metrics, LLM-as-judge, and benchmarks. Use when testing prompt quality, validating RAG pipelines, measuring safety (hallucinations, bias), or comparing models for production deployment.
How do I install it?
Run `npx skills add majiayu000/claude-skill-registry --skill evaluating-llms --agent claude-code` — it drops the skill into your project so the agent can pick it up. Swap the --agent value for codex, cursor or copilot if you use one of those.
Where does this skill come from?
From majiayu000/claude-skill-registry, a repository with 534 stars. We read it straight from the repository tree rather than a submitted listing, so what you see here is what is actually published.
Is a popular skill a good skill?
Not necessarily. Stars measure attention, not adoption — a repository can trend for a week and be abandoned. That is why we show the weekly change from our own snapshots next to the total, instead of a single flattering number.