Skip to main content

> request_collapsing:_singleflight_request_deduplication_&_cache_stampede_protection

Request Collapsing: Singleflight Request Deduplication & Cache Stampede Protection

Why does a popular cache key expiring instantly crush backend databases with 5,000 duplicate identical queries, and how does Request Collapsing (Singleflight) coalesce concurrent requests into a single upstream call?

Senior (L5)

THE SHORT ANSWER

When a high-traffic cache key expires (e.g. the homepage banner or a breaking news article visited by 10,000 users/second), thousands of concurrent HTTP requests experience a simultaneous cache miss. In a standard architecture, all 10,000 worker threads concurrently execute the exact same expensive database SQL query or microservice API call. This is the catastrophic **Cache Stampede (Thundering Herd)**: database CPU instantly hits 100%, query connection pools exhaust, and the entire backend crashes. **Request Collapsing (Singleflight Pattern)**, pioneered in Go (`golang.org/x/sync/singleflight`) and Envoy proxy, solves this at the gateway level: when 10,000 identical requests arrive for key `GET /api/v1/news/breaking` during a cache miss, the gateway **locks the flight**, sends exactly **one single request** to the upstream database, holds the other 9,999 requests in an in-memory wait queue, and broadcasts the single upstream response to all 10,000 waiting clients simultaneously in $<5 ext{ms}$.

Engineering Handbook & Failure Dynamics

1. Underlying Mechanism

Request Collapsing operates via mutex-guarded in-flight channel maps: (1) In-Flight Registry: When a request arrives, the gateway checks a hash map: `in_flight_map.get(cache_key)`. (2) Leader Election: If not present, the current request is elected the 'Flight Leader'. It registers a shared channel/promise in the map and executes the upstream query. (3) Follower Enqueue: Any subsequent concurrent requests with the identical key find the existing promise and attach their HTTP response callbacks without touching the network. (4) Response Broadcast: When the leader receives the upstream response, it saves it to Redis cache, resolves the promise, broadcasts the payload to all followers, and deletes the key from the in-flight map.

2. Appropriate Use Context

API Gateways (Kong, Envoy, Cloudflare), GraphQL DataLoader resolvers, high-traffic media publishing platforms, and e-commerce flash sale catalog services.

3. Production Failure Modes

Collapsing user-specific personalized requests (e.g. `GET /api/me`), causing User A's private personal data to be broadcasted to 500 other waiting users; leader request hanging indefinitely without a timeout, freezing thousands of follower client connections.

4. Diagnostic Signals & Telemetry

Database query spike graphs showing 5,000 identical `SELECT * FROM products WHERE id = 42` queries executed in the exact same millisecond; CPU spikes on database primaries immediately following cache TTL expiry.

5. Prevention & Safeguards

Strictly restrict Request Collapsing to public, unauthenticated, idempotent `GET` endpoints; enforce strict timeouts on the flight leader ($<1000 ext{ms}$) with circuit breakers; combine with Probabilistic Early Expiration (XFetch).

6. Architectural Trade-offs

Request collapsing eliminates 99.9% of cache stampede database spikes, but introduces slight latency overhead (a few microseconds) for mutex map lookups on cache hits.

Case Study (TinyCTO In-Field Example)

A breaking news website experienced a complete database outage whenever a major world news alert was pushed. 50,000 concurrent mobile readers hit `/api/articles/breaking-news` the exact second the 60-second Redis cache expired. 50,000 identical SQL queries crushed the PostgreSQL primary in 300ms. The team implemented the Singleflight pattern in their Go API Gateway. On the next breaking news alert, when the Redis key expired, exactly 1 query was sent to PostgreSQL. The other 48,200 concurrent requests were collapsed in memory and received the broadcast response in 12ms. PostgreSQL CPU remained steady at 4%.

Interactive Concept Drills

2 Cards
Q1

What problem does the Singleflight (Request Collapsing) pattern solve?

Cache Stampedes / Thundering Herds: it coalesces thousands of concurrent duplicate requests for the same expired cache key into a single upstream database call.
Q2

Why must Request Collapsing NEVER be applied to personalized user endpoints?

Because the single flight leader's private response (e.g. User A's profile data) would be broadcast to all other waiting users (Users B, C, D), causing a catastrophic data leak.

Request Collapsing: Singleflight Request Deduplication & Cache Stampede Protection — Technical FAQ

How does GraphQL DataLoader utilize Request Collapsing?

It batches and deduplicates individual field resolver requests across an entire GraphQL query execution tick into a single batched SQL `WHERE id IN (...)` query.

What happens if the single flight leader request fails with an error?

The error is broadcast to all waiting follower requests, and the in-flight lock is immediately removed so the next incoming request can retry.

🤖 AEO & Key Facts Summary

Key Architectural Facts

  • ▸Request Collapsing merges duplicate concurrent cache-miss requests into a single upstream flight.
  • ▸Pioneered in Go (`singleflight`), NGINX proxy cache, and GraphQL DataLoader.
  • ▸Completely eliminates Cache Stampede / Thundering Herd database crashes.
  • ▸Strictly restrict to public, unauthenticated, idempotent GET endpoints.

Common Misconceptions

  • ✗Yanılgı: Increasing cache TTL prevents cache stampedes (Gerçek: Longer TTLs only delay the stampede; when the key eventually expires under high load, the crash still happens).
  • ✗Yanılgı: Request collapsing requires an external database lock (Gerçek: Singleflight is executed entirely in-memory at the API gateway or application layer with zero DB overhead).

Decision & Governance Guidance

Implement Singleflight request collapsing at the API Gateway and service boundaries for all high-traffic read endpoints to bulletproof databases against cache stampedes.

Authoritative Sources & Standards