executing-distributed-system-tests

What it does

Executes a previously designed distributed systems test plan against a real or simulated cluster. It discovers existing test tooling, validates environmental capabilities, sets up a session directory, and runs test scenarios in plan order. It captures nemesis landing evidence per scenario, runs green-but-broken and weak-oracle audits before any PASS, and handles boundary/fairness scenarios with per-arm verdicts. It also supports author mode to generate skeleton tests and enforces a strict workflow to avoid green-but-broken results becoming passes.

How it works

1. Load the plan from a supplied plan file or extract scenario lists from conversation.
2. Discover the SUT toolbox by scanning the repository for drivers, workload generators, cluster bring-up scripts, tests, docs, andMakefile/targets; record findings under "Toolbox discovered" in the session log.
3. Probe environment capability and guide install if needed: list capabilities required, ask the user what is available and what to skip, then verify capabilities with quick probes (container runtime, toolchains, backends, fault-injection facilities, kernel features, observability).
4. Establish a session directory at a path like {{session_root}}/{{plan_slug}}/{{UTC_timestamp}}/ with subdirectories logs, metrics, artifacts, findings, and a findings template header.
5. Run scenarios in plan order: ensure preconditions, start workload, inject fault, capture the nemesis landing signal, stop and collect, apply oracle, and record the verdict with evidence. For boundary/fairness arms, treat each arm as its own scenario and compute per-arm verdicts.
6. Enforce budget-tier gating: map PASS verdicts to the corresponding budget tier actually met and record the met tier in the session log.
7. On failure, produce reproducer, reduction classification, taxonomy classifications, and evidence before moving on.
8. Before declaring PASS, run green-but-broken checks and weak-oracle audits, recording their results. Only then record the final verdict in findings.

When to use it

Use when running a previously designed distributed-systems test plan against a real or simulated cluster, or when asked to execute the plan, reproduce a distributed bug, run stability tests, drive chaos, validate a release end-to-end, run tenant isolation tests, or when a plan file exists at a caller-specified location and needs to be run.

What it can touch

The skill relies on a toolbox discovered in the repository (drivers, workload generators, cluster bring-up scripts, tests, docs, and CI definitions). It records capability discoveries and uses them to guide execution. The exact tools are discovered and logged under "Toolbox discovered"; execution uses the discovered drivers and harnesses without reinventing them.

Caveats

The skill requires user consent for environment provisioning when capabilities are missing or non-trivial to install. INCONCLUSIVE results may be produced if required capabilities are unavailable or if signals for nemesis landing are ambiguous. The process enforces audit trails: reproducer, reduction classification, taxonomy classification, evidence, and green-but-broken checks before declaring PASS-hardening or other verdicts.