llm-evaluation

What it does

Plans and describes a framework for evaluating LLM applications through automated metrics, human evaluation, and LLM-as-judge methods. It covers measurement of text generation, classification, and retrieval metrics, plus human dimensions (accuracy, coherence, relevance, fluency, safety, usefulness) and LLM-based judging approaches (pointwise, pairwise, reference-based, reference-free). It also provides an outline for quick-start usage, concrete metric implementations (BLEU, ROUGE, BERTScore) and several custom metrics, and guidance for running A/B tests, inter-rater agreement, and regression checks.

How it works

• Defines three core evaluation types: automated metrics, human evaluation, and LLM-as-Judge approaches. For automated metrics, it lists specific metrics per task (Text Generation, Classification, Retrieval) such as BLEU, ROUGE, METEOR, BERTScore, Perplexity, Accuracy, Precision/Recall/F1, Confusion Matrix, AUC-ROC, MRR, NDCG, Precision@K, Recall@K.
• Specifies human evaluation dimensions: Accuracy, Coherence, Relevance, Fluency, Safety, Helpfulness.
• Describes LLM-as-Judge patterns with examples: Pointwise, Pairwise, Reference-based, Reference-free.
• Provides a Quick Start code block showing how to structure metrics, an EvaluationSuite, and how to run evaluation against test cases.
• Gives concrete code snippets for implementing Automated Metrics (BLEU, ROUGE, BERTScore) and Custom Metrics (groundedness, toxicity, factuality) and for LLM-as-Judge and Reference-Based Evaluation prompts and usage.
• Details Human Evaluation Frameworks including Annotations and Inter-Rater Agreement, plus A/B Testing statistical framework with T-tests, Cohen's d, and significance flags.
• Includes a section on Regression Testing with a RegressionDetector that checks for metric regressions against a baseline using a threshold.
• Mentions integration with LangSmith evaluation tooling and example evaluators (qa, context_qa, cot_qa).

When to use it

• When measuring LLM application performance systematically
• When comparing different models or prompts
• When detecting performance regressions before deployment
• When validating improvements from prompt changes
• When building confidence in production systems
• When establishing baselines and tracking progress over time
• When debugging unexpected model behavior

What it can touch

• Tools: claude-code is declared in the skill description.

Caveats

• The material presents example code and evaluation architectures; exact outcomes depend on implementation and data quality. The skill outlines methods and references to external libraries (e.g., nltk, rouge_score, bert_score, transformers, detoxify, scipy) but does not guarantee results or provide end-to-end runtime guarantees. License noted as MIT in the skill metadata.