evaluating-llms-harness
Evaluates LLMs across 60+ academic benchmarks (MMLU, HumanEval, GSM8K, TruthfulQA, HellaSwag). Use when benchmarking model quality, comparing models, reporting academic results, or tracking training progress. Industry standard used by EleutherAI, HuggingFace, and major labs. Supports HuggingFace, vLLM, APIs.
npx skills add OpenRaiser/NanoResearch --skill lm-evaluation-harness --agent claude-code
Same command for any agent — swap --agent for codex, cursor, copilot.
Weekly change comes from our own snapshots, not the repository page — it measures attention, not adoption.
# lm-evaluation-harness - LLM Benchmarking ## Quick start lm-evaluation-harness evaluates LLMs across 60+ academic benchmarks using standardized prompts and metrics. **Installation**: ```bash pip install lm-eval ``` **Evaluate any HuggingFace model**: ```bash lm_eval --model hf \ --model_args pretrained=meta-llama/Llama-2-7b-hf \ --tasks mmlu,gsm8k,hellaswag \ --device cuda:0 \ --batch_size 8 ``` **View available tasks**: ```bash lm_eval --tasks list ``` ## Common workflows ### Workflow 1: Standard benchmark evaluation Evaluate model on core benchmarks (MMLU, GSM8K, HumanEval). Copy this checklist: ``` Benchmark Evaluation: - [ ] Step 1: Choose benchmark suite - [ ] Step 2: Configure model - [ ] Step 3: Run evaluation - [ ] Step 4: Analyze results ``` **Step 1: Choose benchmark suite** **Core reasoning benchmarks**: - **MMLU** (Massive Multitask Language Understanding) - 57 subjects, multiple choice - **GSM8K** - Grade school math word problems - **HellaSwag** - Common sense reasoning - **TruthfulQA** - Truthfulness and factuality - **ARC** (AI2 Reasoning Challenge) - Science questions **Code benchmarks**: - **HumanEval** - Python code generation (164 problems) - **MBPP** (Mostly B
- Quick start
- Common workflows
- Workflow 1: Standard benchmark evaluation
- Workflow 2: Track training progress
- Workflow 3: Compare multiple models
- Workflow 4: Evaluate with vLLM (faster inference)
- When to use vs alternatives
- Common issues
- Advanced topics
- Hardware requirements
- Resources
pip install lm-eval lm_eval --model hf \ lm_eval --tasks list Full MMLU evaluation (57 subjects) Multiple benchmarks at once eval_checkpoint.sh models.txt meta-llama/Llama-2-7b-hf meta-llama/Llama-2-13b-hf mistralai/Mistral-7B-v0.1
What does the evaluating-llms-harness skill do?
Evaluates LLMs across 60+ academic benchmarks (MMLU, HumanEval, GSM8K, TruthfulQA, HellaSwag). Use when benchmarking model quality, comparing models, reporting academic results, or tracking training progress. Industry standard used by EleutherAI, HuggingFace, and major labs. Supports HuggingFace, vLLM, APIs.
How do I install it?
Run `npx skills add OpenRaiser/NanoResearch --skill lm-evaluation-harness --agent claude-code` — it drops the skill into your project so the agent can pick it up. Swap the --agent value for codex, cursor or copilot if you use one of those.
Where does this skill come from?
From OpenRaiser/NanoResearch, a repository with 1,480 stars. We read it straight from the repository tree rather than a submitted listing, so what you see here is what is actually published.
Is a popular skill a good skill?
Not necessarily. Stars measure attention, not adoption — a repository can trend for a week and be abandoned. That is why we show the weekly change from our own snapshots next to the total, instead of a single flattering number.