agentsop-regression-gate
Build a held-out eval set, run it on every prompt/model change, and block regressions in CI. An LM change is a code change — gate it with a test suite (eval set + metric + threshold). Cross-framework SOP not surfaced by any single base skill.
npx skills add agentsope/SkillAlchemy --skill agentsop-regression-gate --agent claude-code
Same command for any agent — swap --agent for codex, cursor, copilot.
Weekly change comes from our own snapshots, not the repository page — it measures attention, not adoption.
What it does
Gated regression protection for language-model changes by building a held-out eval set, selecting a metric from a design repository, and enforcing a threshold in CI so that any prompt/model/retriever change that regresses performance is blocked. It treats an LM change as a code change and requires a test suite (eval set + metric + threshold) to gate commits. It also lays out procedures for building, curating, and deploying the eval loop, with multiple operational steps and safeguards for flakiness and per-slice analysis.
How it works
- Activation: enabled when there is a production-facing change such as a prompt edit, a model swap, a retriever change, or a new CI setup, provided a metric exists or can be designed via the metric design skill.
- Build eval set: generate candidates from the corpus, curate them into a golden set, and freeze with versioning.
- Split: partition into train, dev, and test; the gate runs on TEST only and test remains sealed from prompt authoring.
- Pick metric: consume a metric from the agentsop-metric-design framework; do not invent metrics locally.
- Set threshold: define absolute floor and/or relative no-regression delta (possibly per-slice).
- Wire into CI: add a CI job that runs on PR changes, computes the metric, compares to baseline, and fails the build if thresholds are breached; include before/after in PR comments.
- Flakiness handling: pin seeds, average across runs, and separate genuine drops from flaky results.
- Operational modules include specific OPs for generating eval candidates, curating the golden set, splitting data, setting thresholds, wiring CI gates, bumping baselines intentionally, stabilizing flaky evals, and slicing the eval set for subpopulation checks.
When to use it
Use when any production-facing change needs to be evaluated against a held-out standard and you want to block regressions automatically in CI. Activate for prompt changes, model changes, retriever/reranker changes, recurring negative user feedback, or when setting up CI for an LLM app with no existing eval job.
What it can touch
- It relies on: a held-out eval set, a metric from agentsop-metric-design, and a CI integration workflow that runs an eval against a PR config. It specifies generation of candidates with DatasetGenerator.from_documents(docs).generate_dataset_from_nodes(num=50) and references prompts or synthetic generation methods, but does not enumerate executable tool invocations beyond those. The gate is wired into CI to compare test scores against a baseline.json and to post a before/after table in PR comments.
Caveats
- Requires a calibrated metric from agentsop-metric-design; using an uncalibrated metric will gate poorly.
- The eval set must be held out and not leaked into prompts to avoid memorization effects.
- The threshold policy (absolute floor, relative no-regression, or per-slice) must be defined and kept in versioned form (baseline.json).
- Flakiness handling is essential; without proper mitigation, gates may flap and disrupt development.
# regression-gate — Eval Set + Metric + Threshold, Wired Into CI > "Every subsequent change must be gated on these numbers." > — Synthesized from [[llamaindex]] Stage 2 (eval loop *before* optimizing) [llamaindex-sop-skill/SKILL.md:114-126] > "Compiled program beats baseline on a *held-out* test set (not the val set used in optimization)." > — [[dspy]] Stage 3 exit criterion [dspy-sop-skill/SKILL.md:101] This is an **enhancement overlay**. The regression-gate SOP exists only as fragments scattered across base skills — [[llamaindex]] `OP-10 EvalLoop` ("gate every change"), [[dspy]] train/dev/test split + metric — and is never assembled as a standalone cross-framework discipline. It is the discipline that turns a one-off eval into a *gate*: a test suite that runs in CI on every prompt/model/retriever change and fails the build on regression. It consumes a metric from `[[agentsop-metric-design]]` and, for domain-specific held-out sets, hands off to `[[agentsop-domain-eval-set]]`. --- ## 1. 何时激活 (When to Activate) Activate when **any** of these is true: - **Any prompt change you want to ship safely**: a prompt edit, a system-message tweak, a few-shot-demo swap is about to merge and you
- 1. 何时激活 (When to Activate)
- 2. 核心心智模型 (Core Mental Model)
- Build the eval loop before you optimize anything
- 3. SOP (Standard Operating Procedure)
- Stage 1 — Build the eval set (generate + curate)
- Stage 2 — Split: train / dev / test
- Stage 3 — Pick the metric (cross-link [[agentsop-metric-design]])
- Stage 4 — Set the threshold
- Stage 5 — Wire into CI
- Stage 6 — Handle flaky evals
- 4. 操作模型 (Operations)
- OP-01 — GenerateEvalCandidates
- OP-02 — CurateGoldenSet
- OP-03 — SplitTrainDevTest
What does the agentsop-regression-gate skill do?
Build a held-out eval set, run it on every prompt/model change, and block regressions in CI. An LM change is a code change — gate it with a test suite (eval set + metric + threshold). Cross-framework SOP not surfaced by any single base skill.
How do I install it?
Run `npx skills add agentsope/SkillAlchemy --skill agentsop-regression-gate --agent claude-code` — it drops the skill into your project so the agent can pick it up. Swap the --agent value for codex, cursor or copilot if you use one of those.
Where does this skill come from?
From agentsope/SkillAlchemy, a repository with 255 stars. We read it straight from the repository tree rather than a submitted listing, so what you see here is what is actually published.
Is a popular skill a good skill?
Not necessarily. Stars measure attention, not adoption — a repository can trend for a week and be abandoned. That is why we show the weekly change from our own snapshots next to the total, instead of a single flattering number.