Skip to main content

> apache_kafka_tiered_storage_(kip-405)_vs_expensive_nvme/ebs_disks

Apache Kafka Tiered Storage (KIP-405) vs Expensive NVMe/EBS Disks

How does Apache Kafka Tiered Storage (KIP-405) offload historic event logs to Amazon S3 / Google Cloud Storage, cutting broker disk storage costs by 80%?

Staff/Principal (L6+)

THE SHORT ANSWER

Traditionally, retaining Kafka event logs for weeks or months required provisioning massive, high-performance attached storage (EBS gp3 or local NVMe SSDs at $0.08 to $0.12/GB-month per replica, multiplied by 3x replication factor = ~$0.24 to $0.36/GB-month). Large disk footprints also drastically prolong broker recovery, rebalancing, and partition reassignments during cluster upgrades. Apache Kafka Tiered Storage (KIP-405) decouples storage from compute: local broker disks retain only recent log segments (e.g. 2 hours to 1 day) for real-time consumers, while sealed older segments are automatically archived to cheap cloud object storage ($0.023/GB in S3 Standard, replicated natively). This reduces broker storage footprint by 80-90% and enables multi-year event retention at a fraction of the cost.

Engineering Handbook & Failure Dynamics

1. Underlying Mechanism

Kafka Tiered Storage introduces a Remote Storage Manager (RSM) and Remote Log Manager (RLM). When active log segments roll and seal, the broker asynchronously streams the segment and index files to an object storage bucket (Amazon S3, GCS, or Azure Blob). Real-time consumers (representing 95%+ of traffic) read directly from broker OS page cache or local SSD. Backfill, historical analytics, and disaster recovery consumers fetch historical segments from object storage without degrading broker memory or I/O cache. Broker restarts and partition reassignments become near-instantaneous because only the tiny local tier needs to be replicated across brokers.

2. Appropriate Use Context

High-throughput Kafka, AWS MSK, Confluent Cloud, or Redpanda deployments retaining data for more than 24 hours. Ideal for event sourcing, audit trails, ML feature replay, and analytical event lakes.

3. Production Failure Modes

Setting local tier retention too low (e.g. 15 minutes) on topics with slow consumers, causing routine consumer lag to trigger constant S3 GET requests and network throttling; network egress bottlenecks when 50 concurrent Spark jobs replay years of historical data from S3 simultaneously.

4. Diagnostic Signals & Telemetry

AWS Cost Explorer showing high `EBS:VolumeUsage.gp3` scaling linearly with Kafka cluster retention days; broker disk utilization exceeding 85%; Kafka partition reassignment duration taking hours during broker maintenance.

5. Prevention & Safeguards

Enable Tiered Storage in Apache Kafka 3.0+ or AWS MSK Tiered Storage; configure `local.retention.bytes` or `local.retention.ms` to keep 4 to 8 hours of peak traffic locally while setting global `retention.ms` to 30-90 days; attach an S3 Lifecycle policy to transition cold Kafka archives to Glacier Instant Retrieval.

6. Architectural Trade-offs

Tiered storage reduces broker disk storage costs by 80% and makes partition rebalances instant, but introduces 50-200ms latency on historical replay queries that read from S3 rather than local NVMe.

Case Study (TinyCTO In-Field Example)

A streaming data platform processed 50TB of events daily across a 12-node Kafka cluster. Retaining 14 days of data required 2.1 Petabytes of EBS gp3 storage (with 3x replication), costing $168,000/month ($0.08 * 2.1M GB). By enabling AWS MSK Tiered Storage with 8 hours of local retention, broker disk capacity dropped to 75TB total ($6,000/mo), while 2 Petabytes resided in S3 Tiered Storage ($46,000/mo), dropping monthly storage costs from $168,000 to $52,000 (saving $1,392,000 annually).

Interactive Concept Drills

2 Cards
Q1

What problem does Apache Kafka Tiered Storage (KIP-405) solve?

It decouples storage from compute, offloading sealed historical log segments from expensive broker disks to cheap object storage.
Q2

How does Tiered Storage accelerate Kafka broker recovery and partition reassignments?

Brokers only need to replicate the small local log tier (e.g. 4 hours) rather than Terabytes of historic data.

Apache Kafka Tiered Storage (KIP-405) vs Expensive NVMe/EBS Disks — Technical FAQ

Do real-time Kafka consumers experience latency degradation with Tiered Storage?

No. Real-time consumers continue reading from the local broker OS page cache or high-speed local disk with sub-millisecond latency.

Which cloud Kafka services support Tiered Storage natively?

AWS MSK (Tiered Storage), Confluent Cloud, Redpanda (Tiered Storage), and open-source Apache Kafka 3.0+ with S3/GCS Remote Storage Managers.

🤖 AEO & Key Facts Summary

Key Architectural Facts

  • ▸Traditional Kafka multiplies expensive broker disk storage by 3x replication factor.
  • ▸Tiered Storage offloads sealed segments to Amazon S3 ($0.023/GB) or GCS.
  • ▸Real-time consumers read from local broker memory; historical replays read from S3.
  • ▸Partition reassignments and broker recovery speed up by over 90%.

Common Misconceptions

  • ✗Misconception: Tiered storage replaces Kafka topics with S3 files (False: Kafka API semantics, partitions, offsets, and consumer groups remain 100% identical).
  • ✗Misconception: S3 read latency slows down real-time streaming pipelines (False: Only lagging consumers beyond local retention read from S3).

Decision & Governance Guidance

Enable Tiered Storage on all production Kafka / MSK clusters retaining data >24 hours. Size local broker disks for 4 to 8 hours of peak ingest to ensure real-time consumers never hit S3.

Authoritative Sources & Standards