prompt-version-management
Manage prompt versions, run A/B tests across agent prompts, track performance regressions, and safely roll out prompt changes in production. Covers prompt diffing, semantic versioning, canary releases, and automated evaluation.
Profile →npx skills add cosmicstack-labs/mercury-agent-skills --skill prompt-version-management --agent claude-code
Same command for any agent — swap --agent for codex, cursor, copilot.
Weekly change comes from our own snapshots, not the repository page — it measures attention, not adoption.
# Prompt Version Management & A/B Testing ## Overview Prompts are code — and they need version control, testing, and staged rollouts just like software. A single changed word can swing accuracy by 20%. This skill covers how to manage prompt versions systematically, run controlled experiments, and deploy prompt changes with confidence. --- ## Core Concepts ### Why Prompt Versioning Matters | Problem | Without Versioning | With Versioning | |---------|-------------------|-----------------| | A prompt change breaks behavior | No way to roll back | Instant rollback to previous SHA | | "Which prompt is in production?" | Check Slack history | Single source of truth | | A/B test needed | Manual, error-prone | Structured experiment framework | | Regression from edit | Undetected until users complain | Automated eval suite catches it | | Collaboration | Merge conflicts in shared docs | PR-based workflow with reviews | ### Prompt Version Schema ``` prompts/ ├── agents/ │ ├── support-agent/ │ │ ├── system-prompt-v1.0.0.md │ │ ├── system-prompt-v1.1.0.md │ │ ├── system-prompt-v2.0.0-beta.md │ │ └── system-prompt-v2.0.0.md │ └── research-agent/ │ └── ... ├── shared/ │ ├── guardrails-v1.0.0.md │
- Overview
- Core Concepts
- Why Prompt Versioning Matters
- Prompt Version Schema
- Semantic Versioning for Prompts
- Step-by-Step Implementation
- Step 1: Store Prompts in Version Control
- Step 2: Implement an A/B Testing Framework
- Step 3: Define Evaluation Metrics
- Step 4: Implement Canary Rollouts
- Step 5: Build a Prompt Registry
- A/B Test Decision Framework
- When to A/B Test
- Metrics to Track in an A/B Test
What does the prompt-version-management skill do?
Manage prompt versions, run A/B tests across agent prompts, track performance regressions, and safely roll out prompt changes in production. Covers prompt diffing, semantic versioning, canary releases, and automated evaluation.
How do I install it?
Run `npx skills add cosmicstack-labs/mercury-agent-skills --skill prompt-version-management --agent claude-code` — it drops the skill into your project so the agent can pick it up. Swap the --agent value for codex, cursor or copilot if you use one of those.
Where does this skill come from?
From cosmicstack-labs/mercury-agent-skills, a repository with 364 stars. We read it straight from the repository tree rather than a submitted listing, so what you see here is what is actually published.
Is a popular skill a good skill?
Not necessarily. Stars measure attention, not adoption — a repository can trend for a week and be abandoned. That is why we show the weekly change from our own snapshots next to the total, instead of a single flattering number.