agentsop-domain-eval-set
Build and govern a 50-200 example domain-specific held-out benchmark sampled from real traffic. Distinct from public benchmarks (MMLU/HumanEval/GSM8K via lm-evaluation-harness) which measure GENERAL capability. Only a held-out domain set predicts whether THIS system works on YOUR data. Collect real examples, label, hold out (never train/prompt on it), size 50-200, version it, refresh on drift.
npx skills add agentsope/SkillAlchemy --skill agentsop-domain-eval-set --agent claude-code
Same command for any agent — swap --agent for codex, cursor, copilot.
Weekly change comes from our own snapshots, not the repository page — it measures attention, not adoption.
What it does
The skill defines a process to create and govern a 50–200 example held-out, domain-specific benchmark drawn from real traffic. It emphasizes labeling with ground truth, sealing a test split, versioning the artifact, and refreshing the data on domain drift. It positions this domain set as the decision gate alongside public benchmarks, ensuring that production suitability is measured on task-specific data rather than general capability.
How it works
The SOP enumerates seven stages:
- Collect from real traffic: sample real inputs, stratify by type/segment/difficulty, over-sample edge cases, build a raw pool at least twice the final target with provenance.
- Label and curate: attach ground truth or pass/fail, have two annotators on a subset, resolve disagreements, drop ambiguous items, record the rubric.
- Hold-out discipline: split into train/dev/test and seal the test split so it is never shown to optimizers or used in prompts or fine-tuning.
- Size for 50–200: target 50–200; larger when needed for per-segment confidence; smaller than 50 is not gated.
- Version the set: freeze as an artifact (eval_v1.jsonl) with a manifest including hash, creation date, and labeling rubric; score changes against the same version.
- Leak audit: diff the held-out set against demos, fine-tune data, and optimizer train data; remove overlap if found.
- Pair with public bench: run public benchmarks for sanity and use the domain-held-out as the decision gate; domain set wins if disagreements. A refresh procedure exists for domain drift, with quarterly cadence or major changes, to add fresh examples and bump versions.
When to use it
Activate when you need quantitative evidence that the model works on your specific data distribution, not just general capability. Use this domain held-out set as the gating signal for deployment decisions where production fit matters more than public benchmark scores.
What it can touch
The skill references tools and artifacts for governance: [[agentsop-regression-gate]], [[lm-evaluation-harness]], and DatasetGenerator (LlamaIndex) for bootstrap development. It specifies the artifacts to produce: eval_v1.jsonl and a version manifest, and a leakage-detection process. It does not specify executing those tools directly inside a single run, but it describes integration points and artifact outputs.
Caveats
The process requires real-domain data with provenance and labeling discipline, two annotators on a subset, and strict hold-out sealing. It emphasizes versioned artifacts and drift-driven refresh; overlap with training data contaminates the hold-out and must be removed.
# domain-eval-set — Your Held-Out Domain Benchmark > "Compiled program beats baseline on a *held-out* test set (not the val set used in optimization)." > — DSPy SOP exit criterion [dspy.ai/learn/optimization/overview/] > "Build the eval loop **before** optimizing anything. Every subsequent change must be gated on these numbers." > — LlamaIndex SOP Stage 2 This is an **ENHANCE overlay** skill. It produces one artifact — a versioned, sealed, human-labeled set of 50–200 examples drawn from *your* domain — that other skills consume: `[[agentsop-regression-gate]]` enforces it on every PR, `[[agentsop-metric-design]]` defines the scoring function applied to each example, and `[[lm-evaluation-harness]]` runs the *complementary* public-capability axis. The core claim: **public benchmarks tell you the model is smart in general; only a held-out domain set tells you it works on your task.** The latter is the one that predicts production. --- ## 1. 何时激活 (When to Activate) Activate when **any** of these is true: - **"Does THIS system work on OUR data?"** — someone is about to ship or trust an LLM/RAG/agent system and the only evidence is vibes, a demo, or a public benchmark number. You need a q
- 1. 何时激活 (When to Activate)
- 2. 核心心智模型 (Core Mental Model)
- 3. SOP (Standard Operating Procedure)
- Stage 1 — Collect from real traffic
- Stage 2 — Label and curate
- Stage 3 — Hold-out discipline (the load-bearing stage)
- Stage 4 — Size for 50–200
- Stage 5 — Version it
- Stage 6 — Leak audit
- Stage 7 — Pair with the public bench, gate on the domain set
- Refresh — when the domain shifts
- 4. 操作模型 (Operations)
- 5. 困境决策案例 (Dilemma Cases)
- Dilemma 1 — "We don't have enough labeled domain data to build a held-out set"
What does the agentsop-domain-eval-set skill do?
Build and govern a 50-200 example domain-specific held-out benchmark sampled from real traffic. Distinct from public benchmarks (MMLU/HumanEval/GSM8K via lm-evaluation-harness) which measure GENERAL capability. Only a held-out domain set predicts whether THIS system works on YOUR data. Collect real examples, label, hold out (never train/prompt on it), size 50-200, version it, refresh on drift.
How do I install it?
Run `npx skills add agentsope/SkillAlchemy --skill agentsop-domain-eval-set --agent claude-code` — it drops the skill into your project so the agent can pick it up. Swap the --agent value for codex, cursor or copilot if you use one of those.
Where does this skill come from?
From agentsope/SkillAlchemy, a repository with 255 stars. We read it straight from the repository tree rather than a submitted listing, so what you see here is what is actually published.
Is a popular skill a good skill?
Not necessarily. Stars measure attention, not adoption — a repository can trend for a week and be abandoned. That is why we show the weekly change from our own snapshots next to the total, instead of a single flattering number.