prompt-regression-suite
Design a regression test suite that catches an LLM feature getting worse when the prompt, model, or context changes. Use when asked to stop prompt changes breaking production, set up golden tests or CI gates for an LLM feature, or test a model/prompt upgrade before shipping it. Produces a golden case set, per-case pass criteria, CI gate thresholds, and a triage protocol for failures. For designing first-time evaluation of a new feature use ai-eval-plan instead.
npx skills add mohitagw15856/pm-claude-skills --skill prompt-regression-suite --agent claude-code
Same command for any agent — swap --agent for codex, cursor, copilot.
Weekly change comes from our own snapshots, not the repository page — it measures attention, not adoption.
# Prompt Regression Suite Skill Every prompt tweak, model upgrade, and context change is a deploy. This skill designs the suite that runs on each one and answers a single question: *did anything that used to work stop working?* ## What This Skill Produces - A **golden case set**: curated inputs with per-case pass criteria - **Scoring methods** per case class (exact, rubric-judge, property checks) - **CI gate thresholds** — what blocks a merge vs. what warns - A **failure triage protocol** — flaky vs. regressed vs. golden-set-wrong ## Required Inputs Ask for (if not already provided): - **The feature and its contract** — what the LLM step receives and must produce - **What has broken before** (or nearly) — past incidents seed the best cases - **Real traffic examples** — 10-20 representative inputs, including ugly ones - **What triggers a run** — prompt edits, model bumps, retrieval changes, all of the above? ## Building the Golden Set Compose the set from four deliberate classes — not a random sample: | Class | Purpose | Share | |---|---|---| | **Core paths** | The 5-10 inputs that represent most real traffic | ~40% | | **Past failures** | Every input that caused a bug, complaint, o
- What This Skill Produces
- Required Inputs
- Building the Golden Set
- Scoring Per Case
- CI Gates
- Failure Triage Protocol
- Output Format
- Prompt Regression Suite: [feature]
- Quality Checks
- Anti-Patterns
What does the prompt-regression-suite skill do?
Design a regression test suite that catches an LLM feature getting worse when the prompt, model, or context changes. Use when asked to stop prompt changes breaking production, set up golden tests or CI gates for an LLM feature, or test a model/prompt upgrade before shipping it. Produces a golden case set, per-case pass criteria, CI gate thresholds, and a triage protocol for failures. For designing first-time evaluation of a new feature use ai-eval-plan instead.
How do I install it?
Run `npx skills add mohitagw15856/pm-claude-skills --skill prompt-regression-suite --agent claude-code` — it drops the skill into your project so the agent can pick it up. Swap the --agent value for codex, cursor or copilot if you use one of those.
Where does this skill come from?
From mohitagw15856/pm-claude-skills, a repository with 1,255 stars. We read it straight from the repository tree rather than a submitted listing, so what you see here is what is actually published.
Is a popular skill a good skill?
Not necessarily. Stars measure attention, not adoption — a repository can trend for a week and be abandoned. That is why we show the weekly change from our own snapshots next to the total, instead of a single flattering number.