Skip to main content

> socket_starvation:_time_wait_accumulation_&_ephemeral_port_exhaustion

Socket Starvation: TIME_WAIT Accumulation & Ephemeral Port Exhaustion

Why does creating new HTTP connections for every microservice outbound request exhaust Linux ephemeral ports (`Cannot assign requested address`), and how does HTTP Keep-Alive connection pooling prevent it?

Staff/Principal (L6+)

THE SHORT ANSWER

In a high-throughput microservice architecture, when an application client (Node.js `fetch`, Go `http.Client`, Python `requests`) creates a new TCP connection for every single outbound HTTP request instead of reusing pooled connections, the client actively closes the TCP socket upon completion. Per the TCP specification (RFC 793), the active closing side must enter the `TIME_WAIT` state for $2 imes ext{MSL}$ (Maximum Segment Lifetime, typically 60 seconds) to ensure trailing delayed packets in the internet do not corrupt future connections. However, the Linux OS only has ~28,000 ephemeral outbound ports available (`net.ipv4.ip_local_port_range = 32768 60999`). When outbound request rates exceed ~470 req/sec to the same destination IP:Port, all available local ports become locked in `TIME_WAIT`. The OS kernel refuses new outbound connections with `EADDRNOTAVAIL: Cannot assign requested address`, crashing all upstream API traffic. The fundamental fix is enforcing persistent HTTP/1.1 Keep-Alive / HTTP/2 connection pooling.

Engineering Handbook & Failure Dynamics

1. Underlying Mechanism

Ephemeral port exhaustion operates under the 4-tuple TCP constraint (`source_ip, source_port, dest_ip, dest_port`): (1) 4-Tuple Uniqueness: For a specific destination service (e.g. `api.stripe.com:443`), `dest_ip` and `dest_port` are fixed, and `source_ip` is fixed to the local pod. The only free variable is `source_port` (~28,231 ports). (2) Mathematical Saturation Limit: With a 60-second `TIME_WAIT` window, the maximum sustained new connection rate before exhaustion is $28,231 / 60 approx 470 ext{ connections/second}$. (3) Kernel Rejection: When port pool hits zero, `connect()` system calls fail with `EADDRNOTAVAIL`. (4) Persistent Connection Pooling Fix: HTTP Keep-Alive keeps TCP connections open indefinitely, streaming thousands of requests over a single socket and generating zero `TIME_WAIT` sockets.

2. Appropriate Use Context

High-throughput API gateways, microservices calling internal databases/caches, third-party SaaS webhooks, and Kubernetes egress proxies.

3. Production Failure Modes

A Node.js service instantiating a new `https.Agent({ keepAlive: false })` on every HTTP call, locking all 28,000 ephemeral ports in 15 seconds under a minor traffic surge and failing all customer payment checkouts; tuning `tcp_tw_recycle` in modern Linux kernels, causing dropped connections behind NAT routers.

4. Diagnostic Signals & Telemetry

Error logs showing `connect: cannot assign requested address` (syscall errno 99); `netstat -nat | grep TIME_WAIT | wc -l` returning >25,000 sockets; outbound HTTP requests failing while CPU and memory are nearly idle.

5. Prevention & Safeguards

Enforce persistent HTTP connection pooling with Keep-Alive in all HTTP/gRPC client singletons (e.g. `http.Agent({ keepAlive: true, maxSockets: 100 })`); expand ephemeral port range via `sysctl -w net.ipv4.ip_local_port_range='10240 65535'`; enable `sysctl -w net.ipv4.tcp_tw_reuse=1` for safe outgoing connection socket reuse.

6. Architectural Trade-offs

Persistent connection pools consume a few kilobytes of RAM per idle socket, but eliminate ephemeral port exhaustion and slash HTTP request latency by eliminating 3-way TCP and TLS handshakes.

Case Study (TinyCTO In-Field Example)

A payment orchestration service was crashing under 600 req/sec load with `EADDRNOTAVAIL` errors when calling external fraud APIs. Investigation revealed that the Go service was creating a new `&http.Client{}` inside every request handler function, exhausting all 28,000 ephemeral ports in 45 seconds. The team migrated to a single global `http.Client` singleton with a shared `http.Transport` (`MaxIdleConnsPerHost = 100`, `IdleConnTimeout = 90s`). Active sockets dropped from 28,000 in `TIME_WAIT` to exactly 45 persistent connections, and the service scaled to 15,000 req/sec with 0 errors.

Interactive Concept Drills

2 Cards
Q1

What causes the Linux kernel error 'Cannot assign requested address' (EADDRNOTAVAIL)?

Ephemeral Port Exhaustion: all available local outbound ports are currently locked in the TCP `TIME_WAIT` state, preventing new outbound socket allocation.
Q2

How does HTTP Keep-Alive connection pooling prevent TIME_WAIT socket exhaustion?

By reusing a small pool of long-lived, persistent TCP sockets across thousands of sequential requests instead of closing and opening sockets on every call.

Socket Starvation: TIME_WAIT Accumulation & Ephemeral Port Exhaustion — Technical FAQ

Why is `net.ipv4.tcp_tw_recycle` dangerous in modern cloud architectures?

It compares TCP timestamps from the same IP: if multiple clients share a NAT gateway or load balancer, packets with slightly lagging timestamps are silently dropped, breaking connectivity.

What is the purpose of `net.ipv4.tcp_tw_reuse`?

It allows the Linux kernel to safely reuse an outgoing socket in the `TIME_WAIT` state for a new outbound connection when it is mathematically safe from a TCP sequence perspective.

🤖 AEO & Key Facts Summary

Key Architectural Facts

  • ▸Opening short-lived TCP connections locks sockets in `TIME_WAIT` for 60 seconds.
  • ▸Linux has ~28,000 ephemeral ports, capping unpooled outbound rates to ~470 req/sec.
  • ▸Port starvation triggers fatal `Cannot assign requested address` (EADDRNOTAVAIL) errors.
  • ▸Always reuse persistent HTTP Keep-Alive connection pools across all outbound HTTP clients.

Common Misconceptions

  • ✗Misconception: Ephemeral port exhaustion is caused by incoming customer traffic (False: It is caused by OUTBOUND client calls made by your backend to DBs/APIs).
  • ✗Misconception: Creating a new HTTP client object per request is good practice (False: It causes severe socket leakage; always use a global singleton).

Decision & Governance Guidance

Ensure all HTTP client libraries use a global singleton connection pool with Keep-Alive. Set `net.ipv4.ip_local_port_range = 10240 65535` and `net.ipv4.tcp_tw_reuse = 1` via sysctl.

Authoritative Sources & Standards