agentsop-test-fix-loop

What it does

Wires a verify-then-fix loop around a code-editing LLM agent. The agent edits, runs lint/test, reads the verifier output, applies fixes, and re-runs, bounded by an iteration cap and escalation rules. Activates when a coder agent has a verifiable success criterion (exit code, type-checker output, failing assertion) and the user wants convergence to green. Works framework-agnostic by wrapping various tooling chains (auto-lint/auto-test, SWE-Bench loop, LangGraph, Claude Code bash tool).

How it works

Follows a workflow with five steps:

• Step 1: Wire the verifier command by selecting a fast-to-slow sequence (format, lint, type, test, build) and binding lint and test commands into one shell command.
• Step 2: Capture stdout, stderr, and exit_code from the verifier run; always capture all streams and set NO_COLOR=1; include a timeout.
• Step 3: Format the verifier feedback into a concise, hard-cap digest (≤2k tokens) highlighting the first failure, last traceback frame, and anchor to the changed file/lines, with a hypothesis.
• Step 4: Bound the iterations with a hard cap (MAX_ITERS) and stall detection (same error twice triggers early exit); commit after each edit.
• Step 5: Per-iteration commit using a structured message: git commit -am "agent[iter N]: <one-line>" to keep an auditable history.
• Step 6: Detect success precisely via signals like exit 0 with no diagnostics, or exit 0 with warnings; handle false positives (no tests, swallowed commands) accordingly.
• Step 7: Escalate or commit on exit, surface the last feedback, and tag the escalation reason (exhausted, stalled, env_failure, timeout).

When to use it

Use when there is a verifiable success command (pytest, ruff, mypy, eslint, tsc, go test, cargo check) and the goal is to have the agent converge to green, including SWE-Bench-style runs or CI guardrails requiring green before completion. Do not use when success is subjective, verifier is slow/interactive, human review gates the process, or edits are exploratory.

What it can touch

The skill relies on tools such as the verifier commands (lint/test), a shell or subprocess interface, and git for per-iteration commits. It references the use of commands like aider --auto-lint --lint-cmd "ruff check ." --auto-test --test-cmd "pytest -x --tb=short" and concrete subprocess invocations in Python illustrating how to capture stdout, stderr, and exit_code. It uses file anchors from git diffs and formatting of verifier messages but does not expose additional external touch points beyond those described commands.

Caveats

The protocol depends on a reliable verifier signal (exit_code or parseable output). It requires an explicit iteration cap (MAX_ITERS) and an escalation path if the loop cannot converge. It emphasizes not feeding raw verifier output; it requires formatting into a focused digest. The approach assumes environment availability for commands and that per-iteration commits are possible; it warns about false positives (e.g., tests skipped, swallowed exit codes). The skill itself licenses as MIT and is framework-agnostic, wrapping various toolchains as described.