> tpl_ops_008
Alert Catalogue, Routing and On-Call Plan
Comprehensive engineering alert catalog, routing matrix, and sustainable on-call rotation schedule defining symptom-based alerting thresholds, severity-to-notification pathways, alert fatigue mitigation policies, and fair on-call compensation models.
SRE alerting framework cataloging symptom-based pages, alert routing rules, escalation paths, and sustainable on-call team compensation.
Important Tech Document Template & Operational Notice
TinyCTO.tv Tech Document Template Notice: This template is a general educational and operational starting point. It is not legal, tax, accounting, investment, procurement, regulatory, security or certification advice. Requirements vary by jurisdiction, organization, contract and risk. Review and adapt it with qualified professionals before relying on it.
Problem Solved
Engineering teams suffer severe alert fatigue from noisy, un-actionable alerts waking engineers at 3 AM for false positives, causing real outages to be ignored and inducing catastrophic team burnout.
When to Use
- •Establishing a new 24/7 or follow-the-sun on-call rotation for distributed engineering teams
- •Auditing and cleaning up alerting rules across Prometheus Alertmanager, Datadog, or PagerDuty
- •Instituting fair compensation, time-off-in-lieu, and handoff protocols for on-call shifts
When NOT to Use
- •For managing a live active Sev-1 outage war room (use TPL-OPS-007)
- •For high-level enterprise architecture telemetry design (use TPL-ARC-013)
5 Template Sections & Structural Outline
Alerting on user-impacting symptoms (e.g. 5xx error rate high, checkout failing) rather than internal causes (e.g. CPU at 85%), eliminating 90% of alert noise.
P1 (Immediate phone call / page to on-call), P2 (Urgent Slack alert during business hours), P3 (Automated Jira ticket), and P4 (Informational metric logging).
Strict rule that every P1/P2 page must include a direct link to an actionable, step-by-step triage runbook and relevant dashboard link.
Maximum 1 week on-call per 6-8 weeks per engineer, handover checklists, and primary/secondary pairing to prevent isolation.
Structuring transparent standby stipends (for carrying the pager) plus hourly pay for active incident resolution, alongside time-off-in-lieu (TOIL).
Completion Instructions
Independent Review Checklist
- All mandatory sections completed
- No secrets or passwords included
- Executive sponsor sign-off obtained
Alert Catalogue, Routing and On-Call Plan - Worked Case Study
Fictional Entity: Sovereign Payments 24/7 Global On-Call Rotation and Alert Modernization
Real-world production case study demonstrating complete operational adoption for Sovereign Payments 24/7 Global On-Call Rotation and Alert Modernization.
- •Eliminated 78% of false-positive pages by transitioning 140 raw infrastructure alerts to 12 symptom-based SLO alerts
- •Instituted a follow-the-sun on-call rotation across London and Singapore, slashing nocturnal pages by 92%
- •Established transparent standby compensation policy adopted company-wide, reducing on-call stress attrition to near-zero
Frequently Asked Questions
What is the Google SRE rule regarding "Alerting on Symptoms vs Causes"?
Causes are what is happening internally (e.g. CPU utilization is 92% or disk IOPS are elevated). Symptoms are what the user is experiencing (e.g. checkout API is returning 500 errors or page load latency is >2 seconds). Paging on causes creates massive noise because systems routinely run at high CPU without impacting users. Page only on symptoms.
What defines a "Healthy On-Call Shift" according to industry standards?
A healthy on-call shift produces no more than two pages per 12-hour period, with zero un-actionable pages. If an engineer is paged multiple times every night, the on-call rotation is broken, and technical debt must be prioritized immediately.
Why must every P1 alert include an automated runbook URL?
When an engineer is awakened at 3:30 AM, their cognitive reasoning is severely impaired by sleep inertia. A clear, step-by-step runbook with verified diagnostic commands and rollback steps prevents panic-induced mistakes that turn a minor glitch into a total outage.
Download Tech Document Pack
Auth RequiredDownload all blank templates, worked scenarios, and verification manifests in a single verified archive.
Authoritative Sources
- Google SRE Workbook: Being On-CallGoogle SRE • OFFICIAL REQUIREMENT
- PagerDuty: Incident Response and On-Call Health GuidePagerDuty • OFFICIAL REQUIREMENT