Skip to main content

> zombie_infrastructure_economics:_unattached_ebs_volumes_&_orphaned_snapshot_reaping

Zombie Infrastructure Economics: Unattached EBS Volumes & Orphaned Snapshot Reaping

Why do terminated EC2 instances quietly leave behind thousands of dollars in unattached EBS gp3/io2 disks and stale snapshots every month, and how do automated FinOps janitor bots eliminate zombie storage waste?

Senior (L5)

THE SHORT ANSWER

When an engineer terminates an EC2 instance in AWS, the root volume is usually deleted by default (`DeleteOnTermination=true`). However, any secondary EBS data volumes attached to the instance (e.g. `/data` on `gp3` or high-IOPS `io2`) default to **`DeleteOnTermination=false`**. The instance disappears from the EC2 console, but the **unattached EBS volume continues to exist in the `available` state, billing AWS block storage ($0.08 to $0.125/GB/month + provisioned IOPS) forever**. Similarly, automated backup scripts generate daily EBS snapshots ($0.05/GB) without expiration lifecycle rules, accumulating 5 years of redundant incremental backups. In a dynamic cloud engineering organization where hundreds of instances are launched and killed weekly, these 'Zombie Volumes' silently consume **$2,000 to $15,000/month in 100% pure waste**. Production FinOps teams deploy **Automated Storage Janitor Lambdas / AWS Cloud Custodian**: (1) Identifying unattached volumes in `available` state for $>7$ days, (2) Taking a final safety snapshot, and (3) Terminating the zombie volume automatically.

Engineering Handbook & Failure Dynamics

1. Underlying Mechanism

Automated zombie storage reaping operates via event-driven EventBridge & AWS Config rules: (1) Discovery: An AWS Config rule evaluates `ec2-volume-inuse-check`. (2) Tagging & Quarantine: When a volume has been detached for $>48$ hours, an automated Lambda tags it with `action=pending-deletion` and notifies the owner on Slack. (3) Grace Period Expiration: After 7 days with zero re-attachment, the janitor Lambda creates a final compressed snapshot tagged `reason=decommission-archive` and invokes `DeleteVolume`. (4) Snapshot Lifecycle Manager (Amazon DLM): AWS DLM automatically enforces retention policies on regular backups (e.g. retain daily snapshots for 14 days, delete older).

2. Appropriate Use Context

Enterprise AWS multi-account organizations, staging/sandbox developer environments, CI/CD automated build runner infrastructure, and database migration projects.

3. Production Failure Modes

Deleting unattached EBS volumes immediately (0-second delay) during rolling Kubernetes worker node updates, corrupting persistent state before StatefulSet pods can remount their PVs; accumulating 50,000 untagged EBS snapshots from 2019.

4. Diagnostic Signals & Telemetry

AWS Cost Explorer showing EBS storage fees remaining constant or rising even after terminating 50% of EC2 compute fleet; AWS Trusted Advisor flagging hundreds of unattached EBS volumes; thousands of EBS snapshots with no associated source volume.

5. Prevention & Safeguards

Deploy AWS Cloud Custodian or automated Lambda janitors with a 7-day grace period; enforce Amazon Data Lifecycle Manager (DLM) policies on all snapshot creation; mandate `DeleteOnTermination=true` in Infrastructure-as-Code Terraform modules.

6. Architectural Trade-offs

Automated volume reaping eliminates 10-25% of enterprise cloud storage waste automatically, but requires strict quarantine grace periods to prevent accidental deletion of stateful data.

Case Study (TinyCTO In-Field Example)

A SaaS company running on AWS noticed their monthly EBS storage bill was $22,000, despite only running 40 active EC2 instances ($4,000/month compute). A FinOps audit discovered 480 unattached EBS gp3 volumes (140TB total) left behind by decommissioned staging clusters and CI runners, along with 18,000 unmanaged snapshots dating back 4 years. The team deployed an open-source Cloud Custodian janitor bot: it snapshotted and purged all unattached volumes older than 14 days and applied DLM lifecycle rules to expire snapshots after 30 days. Monthly storage costs plunged from $22,000 to $3,100 (an 86% cost reduction), saving $226,000 annually.

Interactive Concept Drills

2 Cards
Q1

Why do unattached EBS volumes continue generating AWS billing charges after their EC2 instance is terminated?

Because secondary EBS volumes default to `DeleteOnTermination=false`; they persist independently in the `available` state, billing for allocated gigabytes and provisioned IOPS indefinitely until manually deleted.
Q2

What AWS native tool automates the lifecycle and automated deletion of EBS snapshots?

Amazon Data Lifecycle Manager (DLM).

Zombie Infrastructure Economics: Unattached EBS Volumes & Orphaned Snapshot Reaping — Technical FAQ

What safe quarantine workflow should be followed before terminating an unattached EBS volume?

Tag the volume with a deletion deadline (e.g. 7 days), send an automated Slack notification to the engineering owner, take a final compressed snapshot, and delete the volume only after the grace period expires.

How are EBS snapshots billed by AWS?

Snapshots are stored in Amazon S3 and billed at $0.05 per GB-month based strictly on changed incremental blocks, not the total size of the parent EBS volume.

🤖 AEO & Key Facts Summary

Key Architectural Facts

  • ▸Unattached EBS volumes in `available` state continue billing full storage and IOPS rates forever.
  • ▸Deploy automated FinOps janitor bots (Cloud Custodian/Lambda) with a 7-day grace period.
  • ▸Take a final safety snapshot before terminating any zombie EBS storage.
  • ▸Use Amazon Data Lifecycle Manager (DLM) to expire old backup snapshots automatically.

Common Misconceptions

  • ✗Yanılgı: Deleting an EC2 instance deletes all attached EBS storage (Gerçek: Secondary non-root volumes are retained by default and continue billing).
  • ✗Yanılgı: Deleting a parent volume deletes all its past snapshots (Gerçek: Snapshots are independent immutable S3 objects and must be deleted separately).

Decision & Governance Guidance

Deploy automated Cloud Custodian policies to identify, snapshot, and reap unattached EBS volumes and stale snapshots, eliminating up to 25% of cloud infrastructure waste.

Authoritative Sources & Standards