> Term
systemic incident
Incidents that arise from the complex interaction of components and historical design choices rather than isolated component failures.
Detailed Explanation
A systemic incident is a cascading failure where no single service technically did anything "wrong" in isolation. Instead, the incident emerges from the interaction between multiple services acting exactly as designed, but leading to an unexpected catastrophic state.
- Normal behavior in Service A triggers defensive behavior in Service B.
- Service B throttles, causing a retry storm from Service C.
- Service C exhausts the database connection pool, taking down the entire application.
These incidents are the hardest to debug because local monitoring usually shows individual components acting correctly up until the moment they are overwhelmed by the systemic feedback loop.
Why It Matters
It exposes the limits of microservice architectures and highlights the need for chaos engineering, bulkheads, and holistic system understanding.
Common Failure Mode
Practical Example
Production Manifestation
A minor configuration change in a non-critical background worker cascades through message queues, causing the primary user authentication service to time out.
Frequently Asked Questions
What is systemic incident in short?
Incidents that arise from the complex interaction of components and historical design choices rather than isolated component failures.
What is the most common failure mode?
Teams spend hours blaming each other because every individual service dashboard shows green, completely missing the toxic interaction happening between the boundaries.
AI Summary
Incidents that arise from the complex interaction of components and historical design choices rather than isolated component failures. It exposes the limits of microservice architectures and highlights the need for chaos engineering, bulkheads, and holistic system understanding.