Root Cause Analysis (RCA) Standard & Causal Methodologies

Every production incident triggering an RCA must be categorized by severity and logged in an authoritative ledger.

Severity Thresholds

• SEV-0 (Catastrophic): Complete outage of primary business capability, active data corruption, security credential leak, or SLA breach penalty exceeding $50,000. Mandates RCA publication within 48 hours and VP-level sign-off.
• SEV-1 (Critical): Core degradation of critical service paths with partial customer impact or degraded redundancy. Mandates RCA completion within 5 business days.
• SEV-2 (Major): Internal tooling disruption, non-critical telemetry loss, or high-potential near-miss. Mandates RCA completion within 10 business days.

Mandatory Document Ledger Roles

1. Incident Commander: Coordinates timeline verification and chairs blameless review ceremony.
2. Lead Investigating SRE: Primary author, telemetry forensics collector, and causal diagram modeler.
3. Owning Service Architect: Validates architectural containment and technical feasibility.
4. Engineering Director: Executive sign-off and permanent budget authorization for CAPA items.

Conflating triggers with root causes is the single most common failure mode in software postmortems. For authoritative lexical taxonomy, consult the RCA Technical Dictionary Definition. This standard establishes four distinct layers:

1. Trigger Mechanism (The Spark): The singular event that initiated the failure sequence (e.g., automated release deployment, sudden customer traffic spike, vendor API timeout).
2. Proximate Cause (The Mechanism): The immediate technical condition directly producing service disruption (e.g., Redis thread pool starvation, TCP SYN backlog queue saturation, deadlocked database transactions).
3. Root Cause (The Systemic Flaw): The fundamental vulnerability in architecture, validation, or automation that allowed the proximate cause to exist and go undetected prior to production release.
4. Contributing Factors (The Environment): Latent environmental, cultural, or observability deficiencies that exacerbated outage duration or degraded response efficiency (e.g., missing metrics, non-existent rollback runbooks, uncoordinated alerting).

RCA reports must not rely on a single narrative technique. All four formal methodologies below are operationalized in the Incident Postmortem & RCA Working Pack (TPL-OPS-002) and benchmarked against real postmortems including INC-001 (The Outage Was Designed Six Meetings Ago), INC-002 (The Cascading Retry Storm), and INC-100 (The System Remembers What the Roadmap Forgot):

A. The 5 Whys (With Branching Guards)

Iterative interrogative technique used to explore cause-and-effect relationships.

• Guard 1: Every 'Why' link must be backed by concrete log/metric timestamps.
• Guard 2: Human error (e.g. 'engineer forgot flag') is explicitly forbidden as a terminal node. The terminal node must reach architectural or process control failure.
• Guard 3: If an answer has multiple independent prerequisites, fork into parallel causal branches.

B. Ishikawa Diagram (Fishbone 6-M System)

Groups contributing conditions into six systemic domains:

1. Methods: Deployment processes, test gate definitions, change approval cadence.
2. Machines / Infrastructure: Cloud compute, memory virtualization, network topology.
3. Materials / Dependencies: Third-party SaaS APIs, open-source packages, base container images.
4. Measurement / Observability: Metrics coverage, log retention, trace sampling, alerting thresholds.
5. Milieu / Environment: Operating load, concurrency spikes, distributed network partitions.
6. Manpower / Organization: On-call fatigue, runbook ambiguities, team communication handoffs.

C. Fault Tree Analysis (FTA)

Deductive top-down Boolean logic analysis tracing undesirable system states back to basic component failures using AND/OR logic gates.

D. Change Analysis (Kepner-Tregoe)

Comparative analysis contrasting:

• What is happening vs What was expected.
• When did it occur vs When did it last succeed.
• Where is it observed vs Where is it absent.

An RCA report is only as valid as the empirical telemetry supporting it. Speculative postmortems are rejected. The forensic evidence checklists from TPL-OPS-002 mandate capturing telemetry during incidents, resolving the evidence gaps observed in cases like INC-065 (The Hypercare Channel Became Permanent) and INC-138 (The Instant Product Required Permanent Hypercare).

Preservation Protocols

1. Snapshotting: Production metrics dashboards (Grafana, Datadog) covering $T_{-2h}$ to $T_{+2h}$ must be permanently archived as immutable images or serialized JSON state.
2. Log Freeze: Raw application and ingress logs must be moved to an append-only, WORM-compliant (Write Once, Read Many) cold storage bucket with SHA-256 integrity verification.
3. Trace Pinning: Exemplar distributed traces demonstrating high-latency tail anomalies must be tagged and exempt from standard rolling TTL deletion.
4. Sanitization: All customer PII, secret keys, and JWT bearer tokens must be masked using non-reversible cryptographic hashes prior to RCA inclusion.

Postmortems often fail because action items decay into neglected backlogs. The 3-Tier SMART CAPA (Corrective and Preventive Action) tracking worksheet and executive presentation template in TPL-OPS-002 operationalize this standard, eliminating the recurring failures analyzed in INC-001 (The Outage Was Designed Six Meetings Ago) and INC-100 (The System Remembers What the Roadmap Forgot):

The SMART Remediation Criteria

Every corrective and preventive action item must satisfy the five SMART engineering criteria before entry into the remediation backlog:

1. Specific: Exactly one architectural boundary, configuration, or operational mechanism is modified. Vague tickets (e.g., 'improve cache resilience') are rejected by the Incident Commander.
2. Measurable: Clear telemetry assertions are defined (e.g., 'p99 latency < 120ms at 5,000 QPS under simulated Redis primary node failure').
3. Achievable: Workable within existing team architectural capacity without requiring undefined platform rewrites.
4. Relevant: Directly addresses an identified causal factor or latent failure condition established in the RCA causal graph.
5. Time-bound: Bound to hard SLA horizons (48h for Tier 1, 30 days for Tier 2, 90 days for Tier 3) tracked in the authoritative engineering ledger.

Remediation Governance & Decay Prevention Mandate

Action item decay is prevented by enforcing non-negotiable governance gates:

• Single Directly Responsible Individual (DRI): Every Tier 1, 2, and 3 action item must be assigned to a named engineer (Remediation Owner), never an amorphous team alias.
• Backlog Priority Enforcement: Tier 2 and Tier 3 tickets are registered as P1 engineering items, automatically bypassing sprint buffer thresholds.
• Weekly Operations Council Review: Unresolved tickets are audited weekly by the Engineering Operations Council until formal verification is complete.
• Mandatory Effectiveness Verification (VoE): A ticket cannot be closed upon code deployment. It requires a mandatory 30-to-90-day verification horizon where production telemetry and automated chaos assertions prove the defect has not recurred.
• Formal Closure Authority: Closure sign-off is restricted to the Incident Commander and VP of Engineering—engineers cannot self-close their own CAPA tickets without independent verification evidence.

An RCA report cannot be approved until all 20 quality criteria are verified:

1. Incident identification and unique tracking reference assigned.
2. Blameless language verified across all narrative sections.
3. Timeline documented down to minute/second resolution with verified UTC timestamps.
4. Detection mechanism identified (automated SLO alert vs customer report).
5. Trigger event clearly isolated from proximate cause.
6. Proximate technical cause supported by empirical metrics or stack traces.
7. 5 Whys analysis reaches architectural or process root causes without blaming human error.
8. Ishikawa fishbone diagram categorizes latent contributing conditions across 6 domains.
9. Fault Tree Analysis or Change Analysis applied for multi-factor failures.
10. Direct financial and customer SLA breach impact quantified.
11. Data integrity verification confirmed (zero silent drops or unhandled corruption).
12. Security and compliance boundaries verified intact.
13. Mitigation and containment steps documented in chronological sequence.
14. Rollback safety and observability gaps formally critiqued.
15. Tier 1 immediate corrective actions verified deployed within 48h.
16. Tier 2 architectural hardening tickets assigned to named engineers with 30-day SLAs.
17. Tier 3 systemic prevention initiatives scoped for sibling services with 90-day SLAs.
18. Raw telemetry snapshots archived in immutable storage.
19. Blameless postmortem review meeting held with engineering stakeholders.
20. Executive engineering leadership sign-off secured and recorded.