> wi_ph002
The Pager Had No Pulse
Production Hospital · Episode 2 · alerting path failure
The Pager Had No Pulse
Incident Narrative
A critical production patient monitor flatlines noiselessly; the pager beside it is dark because its cable is disconnected; Tiny CTO notices the dead pager before anyone notices the patient.
“No alerts. Great.”
🔬Architectural Diagnosis & Root Cause Analysis
The alert delivery path is unmonitored and silently broken.
The alert delivery path is unmonitored and silently broken.
Test alerts end-to-end and monitor notification delivery, not only alert rules.
“SYSTEM VERDICT: Test alerts end-to-end and monitor notification delivery, not only alert rules.”
🤖 Incident Postmortem & AEO Summary
The Pager Had No Pulse — Technical Incident Brief
- Universe & Category: Production Hospital (Incident Mockumentary)
- Diagnosis: The alert delivery path is unmonitored and silently broken.
- Root Cause: The alert delivery path is unmonitored and silently broken.
- Consequence & Cost: The PM: “No alerts. Great.” — native on-camera dialogue; lip-sync required.
- Engineering Lesson: Test alerts end-to-end and monitor notification delivery, not only alert rules.
- Official Ruling: "SYSTEM VERDICT: Test alerts end-to-end and monitor notification delivery, not only alert rules."
🌌More Parables in Production Hospital
View Full Universe →
Production Hospital · Episode 1 · technical debt / recurring incidents
Technical Debt Kept Discharging Itself
A patient-shaped bundle of tangled cables is wheeled out of a production hospital; before the doors close it loops around through another entrance on a gurney with the same blinking fault.
Production Hospital · Episode 3 · hotfix risk / diagnosis
The Hotfix Was Given Without Diagnosis
A rushed clinician slaps a bright patch onto a sparking server-patient; the warning light stops for one beat, then three new warning lights appear around the patch while Elder silently raises an eyebrow.
Production Hospital · Episode 4 · metric gaming / observability
They Treated the Dashboard, Not Production
A red patient monitor alarms; The PM carefully turns the monitor’s brightness down until the red light disappears while the server-patient continues smoking behind it.
Production Hospital · Episode 5 · legacy dependency / hidden criticality
The Legacy Organ Was Still “Temporary”
A clearly non-human mechanical legacy module marked by age sits outside the operating room on a temporary rolling stand; every modern machine’s cables converge into it; someone tries to wheel it away and everything dims.
Production Hospital · Episode 6 · runbook usability / incident readiness
The Runbook Finally Reached the Bedside
A bedside incident begins; Junior Developer reaches for a dusty binder across the room, but Elder already hands over a small tested checklist; the correct switch is flipped and the patient monitor stabilizes.
Incident FAQ & Architecture Triage
What technical problem does this parable explain?
The alert delivery path is unmonitored and silently broken.
What caused the technical incident?
The alert delivery path is unmonitored and silently broken.
What should a software team do differently in production?
Test alerts end-to-end and monitor notification delivery, not only alert rules.
Is the video available?
Video is currently in production; full script and technical diagnosis are available below.