prompt-lab
Iterate on LLM prompts with structured evaluation and self-correction. Test prompts against ground truth, compare models, track version history. Self-correction loop sends invalid outputs back to LLM for fixing. Supports both taxonomy classification and QRA (Question-Reasoning-Answer) generation.
npx skills add majiayu000/claude-skill-registry --skill prompt-lab --agent claude-code
Same command for any agent — swap --agent for codex, cursor, copilot.
Weekly change comes from our own snapshots, not the repository page — it measures attention, not adoption.
# Prompt Lab Systematic prompt engineering with ground truth evaluation and **self-correction loop**. ## Key Feature: Self-Correction Loop Unlike simple validation that silently filters invalid outputs, this skill: 1. **Presents vocabulary in prompt** - LLM knows valid options upfront 2. **Validates response** - Pydantic catches invalid tags 3. **Sends correction back to LLM** - "You used invalid tags X. Valid options are Y. Please fix." 4. **Tracks correction rounds** - Metrics show how often LLM needed help This gives the model a chance to self-correct rather than silently failing. ## Quick Start ```bash cd /home/graham/workspace/experiments/pi-mono/.pi/skills/prompt-lab # Find the smallest model that works (NEW!) ./run.sh find-minimum --ground-truth queryspec.json --threshold 0.80 # Run evaluation with self-correction enabled (default) ./run.sh eval --prompt taxonomy_v1 --model deepseek # Compare multiple models on same prompt ./run.sh compare --prompt taxonomy_v1 --models "deepseek,gpt-4o" # View evaluation history ./run.sh history --prompt taxonomy_v1 ``` ## Architecture ``` ┌─────────────────────────────────────────────────────────────┐ │ Stage 1: LLM Call with Vocabulary in
- Key Feature: Self-Correction Loop
- Quick Start
- Architecture
- Commands
- find-minimum - Find Smallest Accurate Model (NEW!)
- eval - Run Evaluation
- compare - Compare Models
- extract-prompts - Extract Prompts from Python
- test-sparta - End-to-End SPARTA QRA Test
- Advanced Usage
- test-sparta Options
- Task-Monitor Integration
- NDJSON Streaming Output
- Validation Features
cd /home/graham/workspace/experiments/pi-mono/.pi/skills/prompt-lab Find the smallest model that works (NEW!) Run evaluation with self-correction enabled (default) Compare multiple models on same prompt View evaluation history Find smallest model for QuerySpec with 80% accuracy threshold Prefer local Ollama models (default) Test Chutes API models instead Run test with task-monitor (enabled by default) View progress in task-monitor TUI
What does the prompt-lab skill do?
Iterate on LLM prompts with structured evaluation and self-correction. Test prompts against ground truth, compare models, track version history. Self-correction loop sends invalid outputs back to LLM for fixing. Supports both taxonomy classification and QRA (Question-Reasoning-Answer) generation.
How do I install it?
Run `npx skills add majiayu000/claude-skill-registry --skill prompt-lab --agent claude-code` — it drops the skill into your project so the agent can pick it up. Swap the --agent value for codex, cursor or copilot if you use one of those.
Where does this skill come from?
From majiayu000/claude-skill-registry, a repository with 534 stars. We read it straight from the repository tree rather than a submitted listing, so what you see here is what is actually published.
Is a popular skill a good skill?
Not necessarily. Stars measure attention, not adoption — a repository can trend for a week and be abandoned. That is why we show the weekly change from our own snapshots next to the total, instead of a single flattering number.