Chapter 5: Storage Tiering, Lifecycle Governance & EBS Rightsizing
Object storage lifecycle cascades, S3 Intelligent-Tiering, incomplete multipart abort, and EBS gp2 to gp3 online migration.
#1. Executive Summary & Problem Statement
Storage costs are insidious because storage is an accumulative asset. Unlike compute, which drops to zero when turned off, unmanaged object storage and unattached block storage volumes continue billing 24/7/365 indefinitely.
Engineering teams routinely commit three major storage blunders:
- Leaving petabytes of old operational logs in Standard S3 storage indefinitely ($0.023/GB/month).
- Leaving provisioned IOPS on EBS gp2 volumes instead of modern gp3 volumes (paying 20% higher base rates).
- Ignoring abandoned incomplete multipart uploads that invisibly accumulate millions of billable storage chunks.
#2. Object Storage Lifecycle Rules & S3 Intelligent-Tiering
Automated lifecycle rules transition data to lower-cost tiers based on access age:
[Day 0] S3 Standard ($0.023/GB) ──> Active access
[Day 30] S3 Standard-Infrequent Access ($0.0125/GB) ──> Infrequent lookups
[Day 90] S3 Glacier Flexible Retrieval ($0.0036/GB) ──> Compliance archive
[Day 365] S3 Glacier Deep Archive ($0.00099/GB) ──> Statutory retention
[Day 2555] Permanent Deletion (7 Years)
Terraform S3 Lifecycle & Multipart Abort Policy
resource "aws_s3_bucket_lifecycle_configuration" "finops_tiering" {
bucket = aws_s3_bucket.data_lake.id
rule {
id = "abort-incomplete-multipart-uploads"
status = "Enabled"
abort_incomplete_multipart_upload {
days_after_initiation = 7
}
}
rule {
id = "intelligent-tiering-and-archive"
status = "Enabled"
transition {
days = 30
storage_class = "INTELLIGENT_TIERING"
}
transition {
days = 90
storage_class = "GLACIER"
}
expiration {
days = 2555 # 7-year regulatory retention
}
}
}
#3. EBS Block Storage: Migrating gp2 to gp3
AWS gp3 volumes provide 3,000 IOPS and 125 MB/s baseline throughput regardless of volume size, at a 20% lower storage cost (0.10/GB for gp2).
Migrating online without downtime using the AWS CLI:
# Query all gp2 volumes in the current region
aws ec2 describe-volumes \
--filters Name=volume-type,Values=gp2 \
--query "Volumes[*].VolumeId" --output text | tr '\t' '\n' | while read vol; do
echo "Modifying volume $vol to gp3..."
aws ec2 modify-volume --volume-id "$vol" --volume-type gp3
done
#4. Unattached EBS Volume Sweeper
Unattached EBS volumes are pure waste. A nightly serverless cron job audits volume attachment status:
# Detect unattached (available) EBS volumes
aws ec2 describe-volumes \
--filters Name=status,Values=available \
--query "Volumes[*].[VolumeId,Size,CreateTime]" --output table
Volumes in available state for days should be snapshotted and purged.
