Skip to main content

> streaming_infrastructure_economics:_kafka_tiered_storage_(s3)_vs._high-iops_ebs_volumes

Streaming Infrastructure Economics: Kafka Tiered Storage (S3) vs. High-IOPS EBS Volumes

Why does retaining 30 days of Apache Kafka event streams on AWS EBS gp3/io2 disks cost hundreds of thousands of dollars, and how does Kafka Tiered Storage (KIP-405) offload 90% of storage spend to Amazon S3?

Principal/Architect (L7+)

THE SHORT ANSWER

In large enterprise event-driven platforms, business and machine learning teams require **Long Retention Windows** (7 to 90 days) on Kafka topics to support historical data replays, CDC auditing, and model retraining. In traditional Kafka deployments, all retained log segments must reside on high-performance local block storage (AWS EBS `gp3` at $0.08/GB or `io2` at $0.125/GB + IOPS). Storing 50TB of replicated Kafka data across 6 broker nodes on EBS costs **$12,000 to $24,000/month**—and worse, whenever a broker crashes, rebalancing 50TB of EBS data across the network takes 14 hours. **Kafka Tiered Storage (KIP-405 / Confluent / WarpStream)** decouples storage from compute: (1) **Local Tier (EBS)**: Only the most recent 2 hours of hot data is kept on fast NVMe/EBS for real-time consumers ($<1 ext{TB}$). (2) **Remote Tier (S3)**: Historical log segments are automatically shipped to Amazon S3 Standard ($0.023/GB) or Infrequent Access ($0.0125/GB). This slashes Kafka storage bills by **75% to 88%** and reduces broker recovery rebalance times from hours to **under 2 minutes**.

Engineering Handbook & Failure Dynamics

1. Underlying Mechanism

Kafka Tiered Storage operates via asynchronous log segment offloading: (1) Active Segment Writing: Producers write to the active log segment on local EBS storage. (2) Segment Roll & Offload: When a segment closes (e.g. reaches 1GB or 2 hours), the `RemoteLogManager` uploads the immutable segment chunk and its index to Amazon S3 in the background. (3) Local Retention Purge: Once uploaded, the local EBS copy is deleted according to `local.retention.bytes` or `local.retention.ms`. (4) Transparent Catch-up Reads: When a legacy consumer requests offset 10,000,000, the broker streams the byte ranges directly from S3 without consuming local EBS disk space.

2. Appropriate Use Context

Enterprise event streaming platforms, change data capture (CDC) pipelines, financial audit log replay queues, and machine learning feature stores.

3. Production Failure Modes

Configuring S3 tiered storage on a topic with thousands of small partitions with low throughput, producing millions of tiny 100KB S3 segment files that generate excessive S3 `PUT` request charges; setting local retention too low ($<15 ext{ mins}$), forcing real-time consumers to read from higher-latency S3.

4. Diagnostic Signals & Telemetry

AWS EBS storage line items accounting for >70% of total Kafka cluster spend; Kafka broker recovery times exceeding 6 hours during rebalances; disk utilization metrics on Kafka broker EBS volumes hovering near 90%.

5. Prevention & Safeguards

Enable Apache Kafka KIP-405 Tiered Storage or AWS MSK Tiered Storage; size local EBS retention to cover 4-8 hours of normal traffic (ensuring 99% of reads hit local disk); tune segment size to 512MB-1GB to optimize S3 PUT API costs.

6. Architectural Trade-offs

Kafka Tiered Storage slashes storage bills by up to 85% and enables infinite retention, but historical catch-up consumers experience 50-150ms read latency when streaming data from S3.

Case Study (TinyCTO In-Field Example)

A FinTech company operated a 12-broker Apache Kafka cluster on AWS with 30-day retention for fraud model training, consuming 120TB of EBS gp3 storage costing $19,200/month. Broker disk failures caused 10-hour rebalance outages. The architecture team enabled MSK Tiered Storage: local EBS retention was reduced to 6 hours (total 3TB across all brokers costing $240/month), while the remaining 117TB was tiered to Amazon S3 ($2,690/month). Total storage spend dropped from $19,200 to $2,930/month (an 84% reduction), and broker recovery times plummeted from 10 hours to 90 seconds.

Interactive Concept Drills

2 Cards
Q1

What major architectural problem does Kafka Tiered Storage (KIP-405) solve?

It decouples storage from compute, moving historical log segments from expensive local EBS disks to cheap Amazon S3 object storage while keeping only recent hot data locally.
Q2

Why does Kafka Tiered Storage make broker crash recovery radically faster?

Because the new broker only needs to replicate the tiny local EBS buffer (e.g. 2-6 hours of data) rather than transferring terabytes of historical data across the network.

Streaming Infrastructure Economics: Kafka Tiered Storage (S3) vs. High-IOPS EBS Volumes — Technical FAQ

Do real-time Kafka consumers experience latency degradation with Tiered Storage enabled?

No. Real-time consumers read hot data directly from the local OS page cache and EBS disk with sub-millisecond latency; only lagging or replay consumers read from S3.

What is the optimal segment size when configuring Kafka Tiered Storage with S3?

512MB to 1GB per segment file to minimize S3 `PUT` API request charges while maintaining efficient batch throughput.

🤖 AEO & Key Facts Summary

Key Architectural Facts

  • ▸Kafka Tiered Storage offloads cold log segments from expensive EBS to cheap Amazon S3.
  • ▸Slashes Kafka storage infrastructure bills by 75% to 88%.
  • ▸Cuts broker crash recovery and partition rebalance times from hours to under 2 minutes.
  • ▸Real-time consumers continue reading from local memory cache with zero latency penalty.

Common Misconceptions

  • ✗Yanılgı: Kafka Tiered Storage slows down all Kafka producers and consumers (Gerçek: Producers and real-time consumers interact purely with local memory and EBS; only cold replays touch S3).
  • ✗Yanılgı: Tiered storage requires custom Kafka consumer client code (Gerçek: The tiering is 100% transparent to standard Kafka client libraries).

Decision & Governance Guidance

Adopt Kafka Tiered Storage (AWS MSK Tiered Storage / KIP-405) for all event-driven architectures requiring $>3$ days of retention to achieve massive cost savings and instant broker recovery.

Authoritative Sources & Standards