gpt-lab
Benchmark and compare small GPTs for task-specific inference. Tests base, fine-tuned, and prompted models against shared eval datasets. Finds minimum viable model, compares fine-tuned vs prompted, and generates reports.
npx skills add majiayu000/claude-skill-registry --skill gpt-lab --agent claude-code
Same command for any agent — swap --agent for codex, cursor, copilot.
Weekly change comes from our own snapshots, not the repository page — it measures attention, not adoption.
# GPT Lab Benchmark and compare small GPTs trained by `/create-gpt` against prompted alternatives. Answers the key question: **"Is fine-tuning worth it for this task?"** ## Quick Start ```bash cd .pi/skills/gpt-lab # Benchmark multiple models on a task ./run.sh benchmark --task qra-validator --models "qwen2.5-0.5b,qwen2.5-1.5b" # Find the smallest model meeting a threshold ./run.sh find-minimum --task qra-validator --threshold 0.85 # Compare fine-tuned vs prompted ./run.sh compare --task qra-validator \ --finetuned ../create-gpt/models/qra-validator/model.gguf \ --prompted deepseek-v3.2 # Profile a single model ./run.sh profile --model ../create-gpt/models/qra-validator/model.gguf --samples 100 # Generate report ./run.sh report --task qra-validator --format markdown ``` ## Commands ```bash ./run.sh benchmark --task NAME --models "model1,model2,..." ./run.sh compare --task NAME --finetuned PATH --prompted MODEL_NAME ./run.sh find-minimum --task NAME --threshold FLOAT ./run.sh profile --model PATH --samples N ./run.sh report --task NAME [--format markdown|json] ./run.sh history --task NAME ``` ## Fine-Tuned vs Prompted Verdict ``` accuracy delta < -5% → NOT_WORTH_IT accuracy delta >=
- Quick Start
- Commands
- Fine-Tuned vs Prompted Verdict
- Integration
cd .pi/skills/gpt-lab Benchmark multiple models on a task Find the smallest model meeting a threshold Compare fine-tuned vs prompted Profile a single model Generate report
What does the gpt-lab skill do?
Benchmark and compare small GPTs for task-specific inference. Tests base, fine-tuned, and prompted models against shared eval datasets. Finds minimum viable model, compares fine-tuned vs prompted, and generates reports.
How do I install it?
Run `npx skills add majiayu000/claude-skill-registry --skill gpt-lab --agent claude-code` — it drops the skill into your project so the agent can pick it up. Swap the --agent value for codex, cursor or copilot if you use one of those.
Where does this skill come from?
From majiayu000/claude-skill-registry, a repository with 534 stars. We read it straight from the repository tree rather than a submitted listing, so what you see here is what is actually published.
Is a popular skill a good skill?
Not necessarily. Stars measure attention, not adoption — a repository can trend for a week and be abandoned. That is why we show the weekly change from our own snapshots next to the total, instead of a single flattering number.