experiment-loop

npx skills add vibeeval/vibecosystem --skill experiment-loop --agent claude-code

Same command for any agent — swap --agent for codex, cursor, copilot.

# Experiment Loop Autonomous, iterative improvement inspired by Karpathy's autoresearch methodology. Define a metric, set a target, and let the loop run until the target is met or the iteration limit is reached. ## The 5-Step Loop ``` 1. HYPOTHESIZE -> Form a specific, falsifiable improvement hypothesis 2. MODIFY -> Apply the minimal code/config/prompt change 3. TEST -> Run the measurement suite (benchmarks, tests, evals) 4. EVALUATE -> Compare result against baseline and previous best 5. DECIDE -> KEEP if better, DISCARD (git stash pop --index) if worse | Repeat until target met OR max_iterations reached ``` Each iteration is atomic: one hypothesis, one change, one measurement, one decision. ## Experiment Definition Define an experiment in your task or in `thoughts/EXPERIMENTS.md`: ```yaml experiment: name: "reduce-api-latency" metric: "p95 response time (ms)" baseline: 340 target: 200 direction: minimize # minimize | maximize max_iterations: 10 # hard cap, never exceed measurement_cmd: "npm run bench:api" measurement_key: "p95" # JSON key from bench output scope: "src/api/" # files the loop is allowed to touch ``` ### Key Fields | Field | Description | |-------|-------------| | `

Save current state
git stash push -u -m "experiment-loop: iteration N baseline"
Run experiment
Decision
if result is better:
git stash drop          # keep changes, discard stash
git stash pop --index   # restore exactly: staged + unstaged