Skip to main content

> socket_starvation:_time_wait_accumulation_&_ephemeral_port_exhaustion

Socket Starvation: TIME_WAIT Accumulation & Ephemeral Port Exhaustion

Why does creating new HTTP connections for every microservice outbound request exhaust Linux ephemeral ports (`Cannot assign requested address`), and how does HTTP Keep-Alive connection pooling prevent it?

Staff/Principal (L6+)

⚡THE SHORT ANSWER

In a high-throughput microservice architecture, when an application client (Node.js fetch, Go http.Client, Python requests) creates a new TCP connection for every single outbound HTTP request instead of reusing pooled connections, the client actively closes the TCP socket upon completion. Per the TCP specification (RFC 793), the active closing side must enter the TIME_WAIT state for 2 imes ext{MSL} (Maximum Segment Lifetime, typically 60 seconds) to ensure trailing delayed packets in the internet do not corrupt future connections. However, the Linux OS only has ~28,000 ephemeral outbound ports available (net.ipv4.ip_local_port_range = 32768 60999). When outbound request rates exceed ~470 req/sec to the same destination IP:Port, all available local ports become locked in TIME_WAIT. The OS kernel refuses new outbound connections with EADDRNOTAVAIL: Cannot assign requested address, crashing all upstream API traffic. The fundamental fix is enforcing persistent HTTP/1.1 Keep-Alive / HTTP/2 connection pooling.

Engineering Handbook & Failure Dynamics

6-Dimensional Architecture Breakdown

⚙️1. Underlying Mechanism

Execution

Ephemeral port exhaustion operates under the 4-tuple TCP constraint (source_ip, source_port, dest_ip, dest_port):

1

4-Tuple Uniqueness: For a specific destination service (e.g. api.stripe.com:443), dest_ip and dest_port are fixed, and source_ip is fixed to the local pod. The only free variable is source_port (~28,231 ports).

2

Mathematical Saturation Limit: With a 60-second TIME_WAIT window, the maximum sustained new connection rate before exhaustion is 28,231 / 60 approx 470 ext{ connections/second}.

3

Kernel Rejection: When port pool hits zero, connect() system calls fail with EADDRNOTAVAIL.

4

Persistent Connection Pooling Fix: HTTP Keep-Alive keeps TCP connections open indefinitely, streaming thousands of requests over a single socket and generating zero TIME_WAIT sockets.

🎯2. Appropriate Use Context

Scope

High-throughput API gateways, microservices calling internal databases/caches, third-party SaaS webhooks, and Kubernetes egress proxies.

⚠️3. Production Failure Modes

P0 Risk
  • ✓

    A Node.js service instantiating a new https.Agent({ keepAlive: false }) on every HTTP call, locking all 28,000 ephemeral ports in 15 seconds under a minor traffic surge and failing all customer payment checkouts

  • ✓

    tuning tcp_tw_recycle in modern Linux kernels, causing dropped connections behind NAT routers

📡4. Diagnostic Signals & Telemetry

Telemetry
  • ✓

    Error logs showing connect: cannot assign requested address (syscall errno 99)

  • ✓

    netstat -nat | grep TIME_WAIT | wc -l returning >25,000 sockets

  • ✓

    outbound HTTP requests failing while CPU and memory are nearly idle

🛡️5. Prevention & Safeguards

Safeguards
  • ✓

    Enforce persistent HTTP connection pooling with Keep-Alive in all HTTP/gRPC client singletons (e.g. http.Agent({ keepAlive: true, maxSockets: 100 }))

  • ✓

    expand ephemeral port range via sysctl -w net.ipv4.ip_local_port_range='10240 65535'

  • ✓

    enable sysctl -w net.ipv4.tcp_tw_reuse=1 for safe outgoing connection socket reuse

⚖️6. Architectural Trade-offs

Trade-off

Persistent connection pools consume a few kilobytes of RAM per idle socket, but eliminate ephemeral port exhaustion and slash HTTP request latency by eliminating 3-way TCP and TLS handshakes.

📋

Case Study (TinyCTO In-Field Example)

REAL-WORLD TELEMETRY

A payment orchestration service was crashing under 600 req/sec load with EADDRNOTAVAIL errors when calling external fraud APIs. Investigation revealed that the Go service was creating a new &http.Client{} inside every request handler function, exhausting all 28,000 ephemeral ports in 45 seconds. The team migrated to a single global http.Client singleton with a shared http.Transport (MaxIdleConnsPerHost = 100, IdleConnTimeout = 90s). Active sockets dropped from 28,000 in TIME_WAIT to exactly 45 persistent connections, and the service scaled to 15,000 req/sec with 0 errors.

Interactive Concept Drills

2 Cards
Q1

What causes the Linux kernel error 'Cannot assign requested address' (EADDRNOTAVAIL)?

Ephemeral Port Exhaustion: all available local outbound ports are currently locked in the TCP `TIME_WAIT` state, preventing new outbound socket allocation.
Q2

How does HTTP Keep-Alive connection pooling prevent TIME_WAIT socket exhaustion?

By reusing a small pool of long-lived, persistent TCP sockets across thousands of sequential requests instead of closing and opening sockets on every call.

Socket Starvation: TIME_WAIT Accumulation & Ephemeral Port Exhaustion — Technical FAQ

Why is `net.ipv4.tcp_tw_recycle` dangerous in modern cloud architectures?

It compares TCP timestamps from the same IP: if multiple clients share a NAT gateway or load balancer, packets with slightly lagging timestamps are silently dropped, breaking connectivity.

What is the purpose of `net.ipv4.tcp_tw_reuse`?

It allows the Linux kernel to safely reuse an outgoing socket in the `TIME_WAIT` state for a new outbound connection when it is mathematically safe from a TCP sequence perspective.

🤖 AEO & Key Facts Summary

Key Architectural Facts

  • ▸

    Opening short-lived TCP connections locks sockets in TIME_WAIT for 60 seconds.

  • ▸

    Linux has ~28,000 ephemeral ports, capping unpooled outbound rates to ~470 req/sec.

  • ▸

    Port starvation triggers fatal Cannot assign requested address (EADDRNOTAVAIL) errors.

  • ▸

    Always reuse persistent HTTP Keep-Alive connection pools across all outbound HTTP clients.

Common Misconceptions

  • ✗

    Misconception: Ephemeral port exhaustion is caused by incoming customer traffic (False: It is caused by OUTBOUND client calls made by your backend to DBs/APIs).

  • ✗

    Misconception: Creating a new HTTP client object per request is good practice (False: It causes severe socket leakage; always use a global singleton).

Decision & Governance Guidance

Ensure all HTTP client libraries use a global singleton connection pool with Keep-Alive. Set net.ipv4.ip_local_port_range = 10240 65535 and net.ipv4.tcp_tw_reuse = 1 via sysctl.

Authoritative Sources & Standards

Technical terms on this page

Throughput