prompt-regression
Use when the user has changed a prompt (system prompt, RAG template, agent instruction, etc.) and wants to know whether the candidate is better or worse than the baseline. Also use when the user mentions prompt A/B testing, prompt comparison, prompt optimization validation, "did my prompt change help," or prompt regression testing. Outputs per-dimension win rates with statistical significance using OpenJudge PairwiseAnalyzer.
Profile →npx skills add agentscope-ai/OpenJudge --skill 06-prompt-regression --agent claude-code
Same command for any agent — swap --agent for codex, cursor, copilot.
Weekly change comes from our own snapshots, not the repository page — it measures attention, not adoption.
<HARD-GATE> NO conclusion about which prompt is better WITHOUT bootstrap 95% CI reported. NO candidate declared "better" WITHOUT position-debiased (swap-aggregate) comparison. NO comparison with fewer than 10 samples per axis — CI is too wide to be meaningful. </HARD-GATE> # Prompt Regression Compare two prompts head-to-head and determine, with statistical rigor, whether the candidate is better, worse, or tied on each evaluation dimension. ## When to Activate - You changed the system prompt and want to verify it's actually better - You're iterating on RAG answer templates - You're optimizing agent step-by-step instructions - You want data to support a prompt change decision ## Checklist You MUST create a task for each item and complete them in order: 1. **Load and analyze prompts** — diff the baseline vs candidate 2. **Derive comparison dimensions** — from the prompt changes + task type 3. **Select graders per dimension** — pairwise, judge, or rule 4. **Run position-debiased comparison** — swap-aggregate to eliminate order bias 5. **Compute statistics** — win rates + bootstrap 95% CI per dimension 6. **Present results** — per-dimension verdict with confidence intervals ## Fast path
- When to Activate
- Checklist
- Fast path: run the bundled script
- Step 1: Load and Analyze Prompts
- Step 2: Derive Comparison Dimensions
- Dimension templates by task type
- Step 3: Select Graders
- Step 4: Run Position-Debiased Comparison
- Pairwise comparison with swap-aggregate
- Step 5: Compute Statistics
- Step 6: Present Results
- Common Mistakes
- Next Skills
python scripts/pairwise.py --comparisons comparisons.jsonl --candidate candidate --baseline baseline
What does the prompt-regression skill do?
Use when the user has changed a prompt (system prompt, RAG template, agent instruction, etc.) and wants to know whether the candidate is better or worse than the baseline. Also use when the user mentions prompt A/B testing, prompt comparison, prompt optimization validation, "did my prompt change help," or prompt regression testing. Outputs per-dimension win rates with statistical significance using OpenJudge PairwiseAnalyzer.
How do I install it?
Run `npx skills add agentscope-ai/OpenJudge --skill 06-prompt-regression --agent claude-code` — it drops the skill into your project so the agent can pick it up. Swap the --agent value for codex, cursor or copilot if you use one of those.
Where does this skill come from?
From agentscope-ai/OpenJudge, a repository with 764 stars. We read it straight from the repository tree rather than a submitted listing, so what you see here is what is actually published.
Is a popular skill a good skill?
Not necessarily. Stars measure attention, not adoption — a repository can trend for a week and be abandoned. That is why we show the weekly change from our own snapshots next to the total, instead of a single flattering number.