Failure Modes Analysis

What it does

Failure Modes Analysis presents a systematic approach to identifying, categorizing, and mitigating potential failures in distributed systems. It explains the distinction between failure modes (what the system does wrong) and failure causes (why it happened), and provides concrete examples and relationships between them. It covers common failure modes such as network partitions, service unavailability, cascading failures, resource exhaustion (CPU, memory, disk, connection pools), database failures, and dependency/configuration issues, followed by a Failure Mode and Effects Analysis (FMEA) framework, severity levels, and detection/mitigation guidance. It also includes templates for documenting failure modes, a health-check and circuit-breaker code snippets, timeouts, error-rate monitoring, and runbooks. The material is organized into sections detailing detection strategies, prevention, and failure-mode documentation, plus SPOF and blast-radius analyses with example content and practical strategies like circuit breakers, bulkheads, degradation, and redundancy concepts. It specifically targets engineers designing resilient distributed systems and incident response planning, using tools like TypeScript pseudocode, markdown templates, and runbook references.

How it works

• Defines Failure Mode as observable wrong behavior and Failure Cause as root reason, showing the cause → mode → impact relationship.
• Lists common failure modes with descriptions, detection signals, and concrete examples (e.g., network partitions, unavailability, cascading failures, resource exhaustion across CPU, memory, disk, and connections).
• Provides an FMEA template and a process (identify component, list failure modes, assign severity/likelihood/detection, calculate RPN, prioritize mitigations).
• Establishes a Severity Classification schema (SEV0 to SEV4) with impact descriptions and response guidance.
• Offers detection techniques (health checks, circuit breakers, timeouts, error-rate monitoring) with code sketches demonstrating how to implement them.
• Includes Failure Mode Documentation templates and a Runbook reference, plus SPOF identification and mitigation strategies (redundancy, geographic distribution, failover, alternatives).
• Supplies Blast Radius analysis and concrete strategies for graceful degradation, bulkheads, and rate limiting to contain impact.

When to use it

• When modeling potential failure scenarios in a distributed system.
• During design reviews to anticipate failure modes and craft mitigations.
• For incident preparation: documenting failure modes and runbooks.
• To guide reliability engineering efforts around detection, mitigation, and prevention.

What it can touch

• Tools and code snippets are shown in the form of TypeScript examples for health checks, circuit breakers, timeouts, and error-rate monitors.
• Mentions and references to runbooks and documentation templates for Failure Mode documentation.

Caveats

• The material provides templates and example content; it does not guarantee real-world coverage of all failure scenarios.
• Contains high-level guidance and illustrative code snippets rather than prescriptive, production-ready implementations.
• License noted as MIT for the skill.