prompt-regression-tester
Compares old vs new prompts across test cases with diff summaries, stability metrics, breakage analysis, and fix suggestions. Use for "prompt testing", "A/B testing prompts", "prompt versioning", or "quality regression".
npx skills add majiayu000/claude-skill-registry --skill prompt-regression-tester --agent claude-code
Same command for any agent — swap --agent for codex, cursor, copilot.
Weekly change comes from our own snapshots, not the repository page — it measures attention, not adoption.
# Prompt Regression Tester Systematically test prompt changes to prevent regressions. ## Test Case Format ```json { "test_cases": [ { "id": "test_001", "input": "Summarize this article", "context": "Article text here...", "expected_behavior": "Concise 2-3 sentence summary", "baseline_output": "Output from v1.0 prompt", "must_include": ["main point", "conclusion"], "must_not_include": ["opinion", "speculation"] } ] } ``` ## Comparison Framework ```python def compare_prompts(old_prompt, new_prompt, test_cases): results = { "test_cases": [], "summary": { "total": len(test_cases), "improvements": 0, "regressions": 0, "unchanged": 0, }, "breakages": [] } for test in test_cases: old_output = llm(old_prompt.format(**test)) new_output = llm(new_prompt.format(**test)) comparison = { "test_id": test["id"], "old_output": old_output, "new_output": new_output, "diff": compute_diff(old_output, new_output), "scores": { "old": score_output(old_output, test), "new": score_output(new_output, test), }, "verdict": classify_change(old_output, new_output, test) } results["test_cases"].append(comparison) results["summary"][comparison["verdict"]] += 1 if comparison["verdict"] == "regressions": results["br
- Test Case Format
- Comparison Framework
- Stability Metrics
- Breakage Analysis
- Fix Suggestions
- Report Generation
- Best Practices
- Output Checklist
What does the prompt-regression-tester skill do?
Compares old vs new prompts across test cases with diff summaries, stability metrics, breakage analysis, and fix suggestions. Use for "prompt testing", "A/B testing prompts", "prompt versioning", or "quality regression".
How do I install it?
Run `npx skills add majiayu000/claude-skill-registry --skill prompt-regression-tester --agent claude-code` — it drops the skill into your project so the agent can pick it up. Swap the --agent value for codex, cursor or copilot if you use one of those.
Where does this skill come from?
From majiayu000/claude-skill-registry, a repository with 534 stars. We read it straight from the repository tree rather than a submitted listing, so what you see here is what is actually published.
Is a popular skill a good skill?
Not necessarily. Stars measure attention, not adoption — a repository can trend for a week and be abandoned. That is why we show the weekly change from our own snapshots next to the total, instead of a single flattering number.