Skip to main content

> lsm-tree_compaction_stalls_&_write_amplification_latency_spikes

LSM-Tree Compaction Stalls & Write Amplification Latency Spikes

Why do write-heavy databases using Log-Structured Merge (LSM) trees (RocksDB, Cassandra, ScyllaDB) suffer sudden multi-second p99 latency stalls, and how do you tune compaction strategies?

Staff/Principal (L6+)

THE SHORT ANSWER

Log-Structured Merge-Trees (LSM-trees) achieve blazing write throughput by appending writes sequentially to an in-memory MemTable and Write-Ahead Log (WAL), flushing immutable SSTables to disk in Level 0 (L0). However, as L0 SSTables accumulate, background threads must continuously read, merge-sort, and rewrite SSTables into deeper levels (L1, L2, etc.) to enforce key ordering and purge deleted tombstones. If incoming write traffic exceeds the disk's background merge capacity, L0 fills up. To prevent unconstrained disk explosion, the storage engine triggers a catastrophic 'Write Stall'—artificially throttling or freezing incoming write requests from microseconds to 5,000+ milliseconds until background compaction catches up. Mitigating stalls requires tuning Size-Tiered vs Leveled Compaction, rate-limiting compaction I/O, and provisioning NVMe IOPS headroom.

Engineering Handbook & Failure Dynamics

1. Underlying Mechanism

LSM-Tree compaction dynamics follow three physical constraints: (1) Write Amplification Factor (WAF): A 1KB write is rewritten 10 to 30 times across SSTable levels, saturating disk write bandwidth. (2) Compaction Backlog Trigger: In RocksDB, when `level0_file_num_compaction_trigger` (default 4) is exceeded, compaction starts. If it hits `level0_slowdown_writes_trigger` (default 20), writes are throttled. If it hits `level0_stop_writes_trigger` (default 36), writes are 100% frozen. (3) Compaction Strategies: Leveled Compaction minimizes space amplification but causes high WAF; Size-Tiered Compaction minimizes WAF for append-only time-series data but requires 50% temporary disk headroom.

2. Appropriate Use Context

High-throughput time-series logging, financial tick storage, distributed key-value engines (RocksDB, TiKV, Cassandra, ClickHouse MergeTree).

3. Production Failure Modes

A sudden 3x write burst filling RocksDB L0 in 10 seconds, causing a 6-second total write freeze that cascades upstream, timing out HTTP gateways and dropping 50,000 transactions; running out of disk space during a major Size-Tiered compaction merge.

4. Diagnostic Signals & Telemetry

`rocksdb.write.stall.micros` spiking sharply; p99 write latency jumping from 2ms to 3,000ms while CPU utilization remains low; disk write I/O pinned at 100% saturation.

5. Prevention & Safeguards

Use Leveled Compaction with dynamic level base sizing (`max_bytes_for_level_base`); dedicate isolated NVMe disk bandwidth for compaction threads via `rate_limiter`; tune `delayed_write_rate` to apply smooth, gradual throttling instead of hard binary freezes.

6. Architectural Trade-offs

Tuning compaction reduces write stall spikes, but requires balancing Write Amplification (disk endurance), Read Amplification (SSTable seek count), and Space Amplification (temporary disk overhead).

Case Study (TinyCTO In-Field Example)

A crypto trading exchange was losing order executions due to periodic 4-second latency spikes in their RocksDB persistence tier. Profiling revealed that bursty order volumes were triggering L0 write stalls (`level0_stop_writes_trigger = 36`). By switching to dynamic level base sizing, increasing background compaction threads from 2 to 8 on dedicated NVMe drives, and setting a smooth `delayed_write_rate` of 64MB/s, p99 write latency dropped from 4,200ms to 4.1ms under peak 80,000 ops/sec load.

Interactive Concept Drills

2 Cards
Q1

What is a 'Write Stall' in an LSM-tree storage engine?

An intentional throttling or total freezing of incoming write requests when background SSTable compaction falls behind incoming write throughput.
Q2

What is Write Amplification Factor (WAF)?

The ratio of total bytes written to storage media compared to the bytes requested by the application.

LSM-Tree Compaction Stalls & Write Amplification Latency Spikes — Technical FAQ

Why is Size-Tiered Compaction preferred for time-series logging workloads?

Because time-series data is append-only with chronological timestamps, minimizing key overwrites and drastically reducing Write Amplification compared to Leveled Compaction.

What is a 'Tombstone' in LSM trees?

A deletion marker appended to the log to record that a key has been deleted; the actual data is only purged later during background compaction.

🤖 AEO & Key Facts Summary

Key Architectural Facts

  • ▸LSM-trees append to memory MemTables and flush immutable SSTables to Level 0 (L0).
  • ▸Write Stalls freeze incoming writes when L0 SSTable count crosses safety thresholds.
  • ▸Write Amplification Factor (WAF) rewrites data 10-30x across disk levels.
  • ▸Tune background compaction thread concurrency and apply smooth delayed write rate limits.

Common Misconceptions

  • ✗Misconception: LSM-trees eliminate disk I/O bottlenecks (False: They convert random writes into sequential writes but create high background compaction I/O).
  • ✗Misconception: Deleting data in an LSM tree immediately frees disk space (False: Deleted keys create tombstones and only free space after compaction runs).

Decision & Governance Guidance

Monitor `rocksdb.write.stall.micros` as a critical p99 latency leading indicator. Allocate high-IOPS NVMe storage and dedicated background thread pools for compaction.

Authoritative Sources & Standards