RigorLoop

What it is

RigorLoop is a statistically-sound agentic loop-engineering framework. You provide a task description, gold-standard input/output examples, and checks; it runs agentic loops that iteratively build and refine a solution and evaluates it using a dev / validation / test split. The produced artifact can be an executable Python script, an agent skill document (SKILL.md), or a guidance file (GUIDANCE.md).

How it works

• Uses a strategy agent directing concurrent executor agents to iteratively build/refine solutions.
• Evaluates results with a strict split into dev, validation, and test sets to ensure honest final scores.
• The final artifact is portable and usable outside RigorLoop (script, skill, or guidance).

Getting started

Install

pip install rigorloop

Quickstart

mkdir my-task && cd my-task
rigorloop init      # scaffolds rigorloop.toml, task.md, examples.jsonl
rigorloop check     # validates everything, prints split sizes, warnings, and the agent-call budget estimate
rigorloop run       # runs the loops; artifacts land in runs/<run_id>/

The final run outputs under runs/<run_id>/final/:

• the solution (solution.py, SKILL.md, or GUIDANCE.md)
• report.md with pass rates and caveats
• test_results.json with the same information in machine-readable form

Getting started (content you provide)

• task.md — plain-language task description.
• examples.jsonl — one JSON object per line with fields input and expected_output.
• Checks — one or more [[checks]] blocks in rigorloop.toml (e.g., json_equality, exact_match).

The knobs (rigorloop.toml)

Key sections include [task], [split], [loop], [validation], [agents], and [[checks]] with options like solution_kind, max_loops, executors_per_loop, max_peeks, cohort_size, patience, target_pass_rate, model, timeout_s.

How it stays honest

• The split is done up front and fingerprinted; resumed runs cannot reshuffle.
• Agents see only dev examples; validation scores are aggregated; test set is evaluated exactly once at the end.
• The final test set is intended to reflect generalization and is evaluated once per run.

Caveats

• The README notes that the final test set is scored exactly once per run; reusing the same examples across runs can bias test results.
• Artifact types include script, SKILL.md, or GUIDANCE.md and transportable artifacts are created in the run directory.

Release history

• v0.2.0 (2026-07-10): Steer the search by the validation champion; add README FAQ; version bump.
• v0.1.1 (2026-07-09): Docs: add AGENTS.md, expand release guide, drop planning docs.
• v0.1.0 (2026-07-08): Full Changelog available in commits for v0.1.0.