eval-design
Use when the user needs to design evaluation datasets, create test cases, stratify samples, generate adversarial examples, extract eval dimensions from traces/specs, or build a labeled evaluation set. Also use when the user mentions test data design, eval coverage, difficulty stratification, synthetic data generation for eval, or "how to create good evaluation data." Outputs datasets in OpenJudge-compatible format.
Profile →npx skills add agentscope-ai/OpenJudge --skill 01-eval-design --agent claude-code
Same command for any agent — swap --agent for codex, cursor, copilot.
Weekly change comes from our own snapshots, not the repository page — it measures attention, not adoption.
# Eval Design Design high-quality evaluation datasets that measure what actually matters for your application. You extract evaluation dimensions from business context, structure them into stratified test cases, and output datasets ready for OpenJudge `GradingRunner`. ## When to Activate - User has agent traces / production logs and wants to build an eval set from them - User has evaluation principles but needs properly stratified test data - User wants to generate adversarial examples that stress-test their system - User needs coverage analysis — are they testing all the right things? - User wants a labeling guide for human annotators ## Checklist You MUST create a task for each item and complete them in order: 1. **Extract eval dimensions** — from traces, spec, or user interview 2. **Design stratified sampling** — 60/30/10 split with difficulty strata 3. **Generate test data** — synthetic inputs + adversarial examples 4. **Output OpenJudge dataset** — structured format ready for GradingRunner ## Coverage check: run the bundled script After you have a dataset, validate coverage with the bundled, tested script (`scripts/coverage_check.py`, standard library only, **no OpenJudge depen
- When to Activate
- Checklist
- Coverage check: run the bundled script
- Step 1: Extract Evaluation Dimensions
- From traces (when user has production data)
- From spec (when user has product docs)
- From interview (when user has neither)
- Output: Test Plan
- Step 2: Design Stratified Sampling
- Difficulty Strata
- Sample Size
- Data Design Quadrants
- Step 3: Generate Test Data
- Synthetic data generation
python scripts/coverage_check.py --dataset eval-data/dataset.jsonl
What does the eval-design skill do?
Use when the user needs to design evaluation datasets, create test cases, stratify samples, generate adversarial examples, extract eval dimensions from traces/specs, or build a labeled evaluation set. Also use when the user mentions test data design, eval coverage, difficulty stratification, synthetic data generation for eval, or "how to create good evaluation data." Outputs datasets in OpenJudge-compatible format.
How do I install it?
Run `npx skills add agentscope-ai/OpenJudge --skill 01-eval-design --agent claude-code` — it drops the skill into your project so the agent can pick it up. Swap the --agent value for codex, cursor or copilot if you use one of those.
Where does this skill come from?
From agentscope-ai/OpenJudge, a repository with 764 stars. We read it straight from the repository tree rather than a submitted listing, so what you see here is what is actually published.
Is a popular skill a good skill?
Not necessarily. Stars measure attention, not adoption — a repository can trend for a week and be abandoned. That is why we show the weekly change from our own snapshots next to the total, instead of a single flattering number.