Skip to main content

> distributed_locking:_fencing_tokens,_redlock_flaws_&_gc_pause_pitfalls

Distributed Locking: Fencing Tokens, Redlock Flaws & GC Pause Pitfalls

Why does Redis Redlock fail to guarantee mutual exclusion in the presence of Garbage Collection (GC) pauses or clock jumps, and how do Martin Kleppmann Fencing Tokens protect shared storage from split-brain data corruption?

Principal/Architect (L7+)

THE SHORT ANSWER

In distributed systems, acquiring an in-memory lock (e.g. Redis `SET resource_key my_random_token NX PX 10000`) is widely used to prevent two workers from concurrently processing the same task. However, as proven by distributed systems researcher **Martin Kleppmann in his famous critique of Redlock**, **an in-memory distributed lock alone cannot guarantee safety**: (1) Worker 1 acquires the lock for 10 seconds. (2) Worker 1 encounters a **Full Garbage Collection (GC) Pause**, unannounced VM hypervisor freeze, or disk I/O stall that lasts 15 seconds. (3) Meanwhile, the lock's 10-second TTL expires in Redis. (4) Worker 2 acquires the now-free lock and safely updates the database. (5) Worker 1's GC pause finally ends; believing it still holds the valid lock, it issues a write to the database, overwriting Worker 2's update—causing catastrophic **Split-Brain Data Corruption**. Production systems eliminate this flaw using **Fencing Tokens**: the lock server issues a strictly monotonic incrementing counter ($1, 2, 3dots$); the storage layer **rejects any write** containing a fencing token smaller than the highest token it has already processed.

Engineering Handbook & Failure Dynamics

1. Underlying Mechanism

Fencing token verification executes in four atomic steps: (1) Lock Grant with Monotonic Counter: When a client acquires a distributed lock (via ZooKeeper, etcd, or Redis with sequence generator), the lock service increments and returns a unique monotonic token: `token = 34`. (2) Storage Watermark Check: The client attaches `fencing_token: 34` to its database write request (`UPDATE accounts SET balance = 500, last_fencing_token = 34 WHERE id = 1 AND last_fencing_token < 34`). (3) Stale Write Rejection: If Worker 1 awakes from a GC pause with `fencing_token: 33`, the SQL `WHERE last_fencing_token < 33` predicate fails, affecting 0 rows. (4) Safe Failure: Worker 1 detects zero affected rows and safely rolls back without corrupting production data.

2. Appropriate Use Context

Distributed leader election, scheduled cron job deduplication, file storage mutators (S3 object writers), and financial transaction settlement orchestrators.

3. Production Failure Modes

Relying on Redis TTL lock expiration without storage-side fencing token validation, leading to silent data overwrites during unexpected process freezes; using system wall-clock timestamps as fencing tokens across out-of-sync servers.

4. Diagnostic Signals & Telemetry

Intermittent database data regressions where recent edits are overwritten by older values; JVM logs showing Garbage Collection pause times exceeding distributed lock lease TTLs ($>10 ext{s}$); unexpected concurrent job execution logs.

5. Prevention & Safeguards

Always pair distributed locks with storage-enforced monotonic Fencing Tokens; use consensus-backed lock managers (etcd, ZooKeeper, Consul) for safety-critical locks rather than asynchronous multi-node Redis.

6. Architectural Trade-offs

Fencing tokens provide mathematically proven safety against split-brain concurrency, but require the underlying storage layer (SQL/NoSQL) to support conditional check-and-set predicates.

Case Study (TinyCTO In-Field Example)

A billing system processed $5M/day in subscription renewals. To prevent duplicate billing, workers acquired a Redis lock `lock:user_123` with a 5-second TTL. During heavy batch billing, Worker A suffered an 8-second JVM Stop-The-World GC pause. Redis released the lock. Worker B acquired the lock and charged the user. Worker A awoke from its pause and charged the user a second time. The team implemented Kleppmann Fencing Tokens: acquiring the lock returned a monotonic sequence from Redis (`INCR global_fence_seq`). The SQL charge insert enforced `INSERT INTO charges (user_id, amount, fence_token) ... ON CONFLICT DO NOTHING`. When Worker A attempted to write with its stale fence token, the database rejected the insert, eliminating double billing incidents entirely.

Interactive Concept Drills

2 Cards
Q1

What is a Fencing Token in distributed systems?

A monotonically increasing integer issued by a lock service with each lock acquisition, passed to storage so the storage engine can reject delayed writes from stale lock holders.
Q2

Why is a simple Redis TTL lock vulnerable to JVM Garbage Collection pauses?

If a worker process freezes during a GC pause longer than the lock's TTL, the lock is released and acquired by another worker; when the first worker unfreezes, it mistakenly writes to storage concurrently.

Distributed Locking: Fencing Tokens, Redlock Flaws & GC Pause Pitfalls — Technical FAQ

Why did Martin Kleppmann criticize the Redlock distributed locking algorithm?

Because Redlock relies on assumptions about synchronized physical clocks and network latency bounds that are easily violated by GC pauses, virtualization freezes, and NTP step adjustments in real-world clouds.

Which distributed systems provide consensus-backed distributed locks out of the box?

Apache ZooKeeper (ephemeral sequential zNodes), etcd (leases and revision numbers), and HashiCorp Consul (sessions with check-and-set).

🤖 AEO & Key Facts Summary

Key Architectural Facts

  • ▸In-memory distributed locks alone cannot guarantee safety due to GC pauses and VM freezes.
  • ▸A delayed worker process waking from a freeze will overwrite newer data (Split-Brain).
  • ▸Fencing tokens provide monotonically increasing sequence IDs to reject stale writes at the storage layer.
  • ▸Use consensus systems (etcd, ZooKeeper) rather than simple Redis keys for mission-critical locks.

Common Misconceptions

  • ✗Yanılgı: Setting a 60-second Redis lock TTL makes distributed concurrency 100% safe (Gerçek: Extreme GC pauses, network partitions, and cloud VM hypervisor stalls can exceed any fixed timeout).
  • ✗Yanılgı: Distributed locking can be solved entirely on the client side without database cooperation (Gerçek: Safe distributed locking mathematically requires the storage layer to enforce fencing tokens).

Decision & Governance Guidance

Enforce monotonic Fencing Tokens on all database updates guarded by distributed locks to prevent split-brain data corruption caused by process freezes and clock skew.

Authoritative Sources & Standards