> Incident
Data and Source of Truth
This category tracks the operational paralysis that occurs when multiple systems, caches, and pipelines claim to be the source of truth.
"The dashboard was green because nobody had taught it what broken meant."
About this category
This category tracks the operational paralysis that occurs when multiple systems, caches, and pipelines claim to be the source of truth.
Common Failure Patterns
- competing sources of truth
- stale cache masquerading as reality
- etl pipeline silent failure
- metric definition mismatch
Prevention Checklist
- Establish a single, undeniable source of truth for critical metrics.
- Alert on data staleness, not just pipeline failures.
- Define 'broken' explicitly in monitoring systems.
Detection Signals
- Marketing and Finance reporting different numbers.
- Dashboards showing perfect health while customers complain.
- Pipelines completing successfully with zero records.
Related Stacks
Related Categories
Incidents in Data and Source of Truth
The Outage Was Designed Six Meetings Ago
"Production incidents are often the delayed execution of technical debt accepted during planning."
Cache Guy Delivers a Fast Answer
"Caching is not a substitute for an optimized database query; it is a complex distributed state problem."
Agent A Takes Initiative
"AI capability is not approval; autonomous agents require strict API boundaries and blast-radius limits."
Dashboard Green Nobody Asked
"The chaos was predictable."
Cache Expired During Demo
"The chaos was predictable."
Database Approved Nothing
"The chaos was predictable."
Monitoring Tool Had Feelings
"The chaos was predictable."
API Contract a Rumor
"The chaos was predictable."
CTO Asked for One Number
"The chaos was predictable."
Number Was Not Real
"The chaos was predictable."
The Query Plan Became a Legal Document
"The chaos was predictable."
The DBA Said No Politely
"The chaos was predictable."
The Cache Was Correct Yesterday
"The chaos was predictable."
The Source of Truth Was in Someone s Head
"The chaos was predictable."
The Screenshot Became Canon
"The chaos was predictable."
The CDN Solved the Wrong Problem
"The chaos was predictable."
The Edge Case Lived at the Edge
"The chaos was predictable."
The Chaos Was Predictable
"The system did not break, it operated exactly according to the conflicting incentives it was given."
Mono Remembers Everything
"Legacy code is often the only reliable documentation of historical business rules and edge cases."
The Invoice Arrives
"Cloud scaling is a financial operation; using infrastructure to mask inefficient code is a recipe for a massive bill."
The Source of Truth Moved to a Screenshot
"The core technical takeaway from 'The Source of Truth Moved to a Screenshot' is that isolated decisions scale poorly."
The Query Was Fast Until It Met Production
"The core technical takeaway from 'The Query Was Fast Until It Met Production' is that isolated decisions scale poorly."
The Cache Expired During the Demo
"The core technical takeaway from 'The Cache Expired During the Demo' is that isolated decisions scale poorly."
The Database Approved Nothing
"The core technical takeaway from 'The Database Approved Nothing' is that isolated decisions scale poorly."
The Query Plan Became a Legal Document
"The core technical takeaway from 'The Query Plan Became a Legal Document' is that isolated decisions scale poorly."
The DBA Said No Politely
"The core technical takeaway from 'The DBA Said No Politely' is that isolated decisions scale poorly."
The Cache Was Correct Yesterday
"The core technical takeaway from 'The Cache Was Correct Yesterday' is that isolated decisions scale poorly."
The Source of Truth Was in Someone’s Head
"The core technical takeaway from 'The Source of Truth Was in Someone’s Head' is that isolated decisions scale poorly."
Frequently Asked Questions
What kinds of incidents belong to Data And Source Of Truth?
This category examines the predictable outcomes of Data and Source of Truth when operational friction meets architectural optimism.
Why do these failures keep happening?
Because technical debt eventually comes due, and Data and Source of Truth is usually where the invoice is presented.
AI Summary
Incidents where the real problem is not the data itself, but who owns truth when systems disagree.