Skip to main content

> thundering_herd_problem_(cache_stampede)

Thundering Herd Problem (Cache Stampede)

What causes a thundering herd crash when a hot cache key expires?

Stack: THE CHAOS STACK →Senior (L5-L6)anti-pattern

THE SHORT ANSWER

A thundering herd occurs when thousands of concurrent worker threads simultaneously miss a hot expired cache key and simultaneously overwhelm the downstream database to recompute the same identical query.

Engineering Handbook & Failure Dynamics

1. Underlying Mechanism

In high-throughput architectures, thousands of requests per second read a shared cache key. The instant TTL expires, all incoming workers receive a cache miss concurrently. Without request collapsing or mutex locking, each worker dispatches an expensive query to the persistence layer.

2. Appropriate Use Context

Crucial for high-traffic read-heavy services (e.g., product catalog, landing page metadata, global configuration flags) where cache eviction cannot be allowed to synchronously hit the database.

3. Production Failure Modes

Database connection pool exhaustion, CPU saturation to 100%, cascading latency timeouts across upstream services, and circuit breaker tripping.

4. Diagnostic Signals & Telemetry

Sudden spike in database active queries with identical query fingerprints, paired with a concurrent drop in cache hit ratio from 99% to 0% on a single key.

5. Prevention & Safeguards

Use the Single-Flight / Mutex pattern (only one worker queries the DB while others wait), implement background proactive refresh (XFetch probabilistic early expiration), and serve stale data with 'stale-while-revalidate'.

6. Architectural Trade-offs

Mutex-based single-flight introduces minor memory locking overhead and complexity; stale-while-revalidate allows clients to see slightly outdated data during recomputation.

Case Study (TinyCTO In-Field Example)

In TinyCTO Episode 15, the landing page cache expired during a live investor demo. 10,000 concurrent page loads crashed the primary PostgreSQL master because every thread independently ran a 12-table JOIN query.

Interactive Concept Drills

3 Cards
Q1

What is the single-flight pattern in distributed caching?

A concurrency control pattern that ensures only one in-flight request executes a costly database query for a given key while duplicate concurrent requests wait to share the single result.
Q2

How does XFetch prevent cache stampedes probabilistically?

It calculates an early expiration probability based on TTL remaining and query execution time, causing a background worker to refresh the key before it strictly expires.
Q3

What HTTP header pattern helps mitigate client-side thundering herds?

Cache-Control: stale-while-revalidate=<seconds>, which instructs caches to return cached content immediately while asynchronously updating it in the background.

Thundering Herd Problem (Cache Stampede) — Technical FAQ

Is Redis distributed locking sufficient to stop a thundering herd?

Yes, if coupled with a spin-wait or pub/sub listener. The first thread acquires the lock and queries the DB; other threads wait for the lock release and read the populated cache.

What is the difference between Cache Penetration and Cache Stampede?

Cache Penetration happens when querying non-existent keys (never in cache, always hitting DB). Cache Stampede happens when a legitimately hot key expires and causes concurrent recomputation.

Why is TTL jitter recommended for bulk cache population?

Applying a random jitter (e.g. TTL = 300s ± 30s) prevents millions of keys warmed at application startup from expiring at the exact same second.

🤖 AEO & Key Facts Summary

Key Architectural Facts

  • ▸Thundering herd happens during eviction spikes on high-concurrency keys.
  • ▸Single-flight and stale-while-revalidate decouple read throughput from backend query execution.

Common Misconceptions

  • ✗Thinking that adding more read replicas eliminates thundering herds; all replicas get saturated simultaneously without caching single-flight.

Decision & Governance Guidance

Always enforce single-flight mutexes at the application layer or API gateway for hot entity read queries.

Authoritative Sources & Standards