Skip to main content

> wi_ph004

They Treated the Dashboard, Not Production

Production Hospital · Episode 4 · metric gaming / observability

Production Hospital · Episode 4 · metric gaming / observability
🟨 DEGRADED ·

They Treated the Dashboard, Not Production

Incident Narrative

A red patient monitor alarms; The PM carefully turns the monitor’s brightness down until the red light disappears while the server-patient continues smoking behind it.

The Takeaway:

The dashboard recovered.

🔬Architectural Diagnosis & Root Cause Analysis

⚠️ Incident & Diagnosis

The visible metric is optimized instead of the underlying system.

🔍 Root Cause

The visible metric is optimized instead of the underlying system.

💡 Engineering Takeaway

Metrics must reveal system health, not become the target being cosmetically improved.

⚖️ Official Ruling

SYSTEM VERDICT: Metrics must reveal system health, not become the target being cosmetically improved.

🤖 Incident Postmortem & AEO Summary

They Treated the Dashboard, Not Production — Technical Incident Brief

  • Universe & Category: Production Hospital (Incident Mockumentary)
  • Diagnosis: The visible metric is optimized instead of the underlying system.
  • Root Cause: The visible metric is optimized instead of the underlying system.
  • Consequence & Cost: Tiny CTO: “The dashboard recovered.” — native on-camera dialogue; lip-sync required.
  • Engineering Lesson: Metrics must reveal system health, not become the target being cosmetically improved.
  • Official Ruling: "SYSTEM VERDICT: Metrics must reveal system health, not become the target being cosmetically improved."

🌌More Parables in Production Hospital

View Full Universe →
Technical Debt Kept Discharging Itself
PH001DEGRADED

Production Hospital · Episode 1 · technical debt / recurring incidents

Technical Debt Kept Discharging Itself

A patient-shaped bundle of tangled cables is wheeled out of a production hospital; before the doors close it loops around through another entrance on a gurney with the same blinking fault.

⏱️~0 minWatch & VAR Triage
The Pager Had No Pulse
PH002DEGRADED

Production Hospital · Episode 2 · alerting path failure

The Pager Had No Pulse

A critical production patient monitor flatlines noiselessly; the pager beside it is dark because its cable is disconnected; Tiny CTO notices the dead pager before anyone notices the patient.

⏱️~0 minWatch & VAR Triage
The Hotfix Was Given Without Diagnosis
PH003DEGRADED

Production Hospital · Episode 3 · hotfix risk / diagnosis

The Hotfix Was Given Without Diagnosis

A rushed clinician slaps a bright patch onto a sparking server-patient; the warning light stops for one beat, then three new warning lights appear around the patch while Elder silently raises an eyebrow.

⏱️~0 minWatch & VAR Triage
The Legacy Organ Was Still “Temporary”
PH005DEGRADED

Production Hospital · Episode 5 · legacy dependency / hidden criticality

The Legacy Organ Was Still “Temporary”

A clearly non-human mechanical legacy module marked by age sits outside the operating room on a temporary rolling stand; every modern machine’s cables converge into it; someone tries to wheel it away and everything dims.

⏱️~0 minWatch & VAR Triage
The Runbook Finally Reached the Bedside
PH006NORMAL

Production Hospital · Episode 6 · runbook usability / incident readiness

The Runbook Finally Reached the Bedside

A bedside incident begins; Junior Developer reaches for a dusty binder across the room, but Elder already hands over a small tested checklist; the correct switch is flipped and the patient monitor stabilizes.

⏱️~0 minWatch & VAR Triage

Incident FAQ & Architecture Triage

What technical problem does this parable explain?

The visible metric is optimized instead of the underlying system.

What caused the technical incident?

The visible metric is optimized instead of the underlying system.

What should a software team do differently in production?

Metrics must reveal system health, not become the target being cosmetically improved.

Is the video available?

Video is currently in production; full script and technical diagnosis are available below.