designing-distributed-system-tests

What it does

This skill prescribes designing test plans for distributed or stateful systems. It instructs you to focus on product claims, generate hypotheses to falsify those claims, and produce a structured plan with change-scoped or project-wide scope. It requires extracting guarantees from multiple sources, enumerating claims, and then selecting techniques and designing executable scenarios with concrete workloads, faults, and checkable oracles. It also mandates documenting environment requirements, mapping coverage, and providing an executable target test file path and a skeleton for tests, all tied to a plan path and scenario identifier.

How it works

• Determine plan scope: choose between Change-scoped (specific change) or Project-wide (system-wide). If ambiguous, ask the user.
• Scope the system by reading project entry points and capturing tenancy, persistence, replication, ordering, retry, observability, and surface boundaries.
• Extract claims and categorize them (Safety, Liveness, Durability, Performance/SLO, Operational, Idempotency/Dedup, Isolation, Ordering, Membership, Boundary, Fairness).
• List missing claims as needed.
• Scope the change or project: identify touched files or enumerate surfaces for project-wide scope.
• Inventory existing tests (project-wide).
• Generate failure-mode hypotheses for each claim, guided by a pitfall catalog and surface categories.
• Select techniques from a catalog and, for serious scenarios, fill the §7.M block with checker patterns from oracle references.
• Map coverage: build a claims-to-hypotheses matrix, identify gaps, and declare environment requirements.
• Design concrete scenarios: specify Workload, Faults, Oracle, Observability, budget tiers, and provide Target test file and Skeleton with AUTO-GENERATED header including plan path and scenario id.
• Ensure scenarios that are serious include §7.M blocks and, if decomposed, §7.M.S arms for boundary or fairness.

When to use it

Applicable when designing a test plan for distributed or stateful systems, especially for persistence, replication, consensus, retries, idempotency, async messaging, multi-tenancy, or partial failure. Use for stability plans, fault matrices, release-validation plans, durability/partition/upgrade/crash-recovery/linearizability/deterministic-simulation plans, tenant isolation plans, namespace isolation plans, fairness/noisy-neighbor plans, or holistic test plans. Trigger even on questions like “what should we be testing” or “are my tenants actually isolated.”

What it can touch

The skill references a curated catalog of techniques and oracle patterns, plus environment requirements. It requires outputs that include a Target test file path and a Skeleton for the test, with an AUTO-GENERATED header identifying the plan path and scenario id. It also expects sections for Workload, Faults, Oracle, Observability, and three budget tiers, plus the plan’s claims and gap mappings.

Caveats

The plan relies on identifying explicit claims from sources such as README guarantees, API docs, architecture/design docs, code comments, and existing tests. Inferred claims may be noted but remain tied to the plan. The workflow emphasizes structured, executable scenarios and traceability between plan and test, including an explicit plan path in the AUTO-GENERATED header.