Skip to main content

> multi-tier_llm_rate_limiting:_tokens-per-minute_(tpm)_&_requests-per-minute_(rpm)

Multi-Tier LLM Rate Limiting: Tokens-Per-Minute (TPM) & Requests-Per-Minute (RPM)

Why do standard API rate limiters fail to prevent upstream LLM provider 429 errors, and how does dual-dimensional TPM/RPM sliding window scheduling smooth AI traffic bursts?

Senior (L5)

THE SHORT ANSWER

Standard API rate limiters (e.g. Nginx or Cloudflare) measure traffic strictly in **Requests Per Minute (RPM)**: treating every incoming HTTP request as an equal unit of load. In Generative AI systems, RPM rate limiting is completely insufficient because LLM providers (OpenAI, Anthropic, Google) enforce strict dual quotas: **Requests Per Minute (RPM)** AND **Tokens Per Minute (TPM)**. A single user sending a 100,000-token PDF analysis request consumes the same RPM as a 20-token greeting, but consumes $5,000 imes$ more TPM quota, instantly triggering upstream HTTP 429 Rate Limit rejections for all other users. Production AI gateways (LiteLLM, Portkey, Envoy AI Gateway) implement **Multi-Tier Dual-Dimensional Token Bucket Scheduling**: (1) Fast Pre-Flight Token Estimation (counting input prompt tokens via `tiktoken`), (2) Atomic Redis Token & Request Leases (reserving both estimated TPM and 1 RPM upfront), and (3) Post-Generation Reconciliation (refunding or settling the exact token delta after streaming finishes).

Engineering Handbook & Failure Dynamics

1. Underlying Mechanism

Dual-dimensional LLM rate limiting executes across four coordinated steps: (1) Prompt Token Sizing: Incoming prompt text is tokenized in-memory via SIMD tokenizer (e.g. `tiktoken` in <0.2ms), estimating total tokens $T_{ ext{est}} = T_{ ext{prompt}} + ext{max_output_tokens}$. (2) Atomic Double Lease (Redis Lua): Atomically checks and decrements both RPM bucket ($R ge 1$) and TPM bucket ($T ge T_{ ext{est}}$). If insufficient TPM, the request is queued in an in-memory priority delay queue rather than rejected. (3) Upstream LLM Execution: The request is forwarded to OpenAI/Anthropic. (4) Reconciliation: Upon stream completion, exact actual tokens $T_{ ext{actual}}$ are calculated from the response metadata, and the difference $T_{ ext{est}} - T_{ ext{actual}}$ is immediately refunded back to the tenant's TPM bucket.

2. Appropriate Use Context

Multi-tenant SaaS LLM gateways, internal developer AI proxy platforms, automated batch document ingestion pipelines, and enterprise AI portals.

3. Production Failure Modes

A single tenant running 50 concurrent automated document summarization threads that exhaust the organization's entire 1,000,000 TPM OpenAI tier-4 limit, causing 100% of executive and customer-facing chatbot queries to fail with HTTP 429 for 3 minutes; static TPM limits that fail to dynamically back off when upstream providers throttle under peak internet congestion.

4. Diagnostic Signals & Telemetry

Upstream provider error logs showing `RateLimitError: Rate limit reached for default-gpt-4o in organization on tokens_per_min`; API gateway metrics showing 0% RPM utilization but 100% TPM exhaustion; client requests timing out during batch workload spikes.

5. Prevention & Safeguards

Deploy an AI gateway proxy (LiteLLM / Envoy) with dual TPM/RPM sliding window limiters; partition TPM quotas strictly across tenants/services (`customer_support: 60%`, `batch_jobs: 20%`, `internal_dev: 20%`); implement automated provider failover routing (e.g. spilling over from OpenAI to Azure OpenAI or Anthropic Claude when TPM exceeds 85%).

6. Architectural Trade-offs

Token pre-flight counting and reconciliation adds ~1-2ms of gateway overhead and requires Redis infrastructure, but completely eliminates cascading upstream HTTP 429 outages.

Case Study (TinyCTO In-Field Example)

A SaaS company with 10,000 active users suffered frequent OpenAI outages during morning peaks because marketing teams launched automated batch copy-generation jobs that burned 800,000 TPM in 10 seconds. The infrastructure team deployed LiteLLM with Dual TPM/RPM Token Buckets and assigned tenant tiers: Marketing Batch was capped at 150,000 TPM with asynchronous delay queues, while Live Customer Chat was guaranteed 600,000 TPM with priority routing. Production HTTP 429 errors dropped from 1,200/day to exactly 0.

Interactive Concept Drills

2 Cards
Q1

Why is traditional Requests-Per-Minute (RPM) rate limiting insufficient for Generative AI APIs?

Because requests have wildly differing payload sizes: a single large document prompt can consume 100,000 Tokens-Per-Minute (TPM), exhausting provider token quotas even when RPM is well below the limit.
Q2

How does Token Reconciliation work in dual-dimensional rate limiters?

The gateway reserves an estimated token count before sending the request; once generation streams to completion, unused reserved tokens are refunded back to the user's quota bucket in Redis.

Multi-Tier LLM Rate Limiting: Tokens-Per-Minute (TPM) & Requests-Per-Minute (RPM) — Technical FAQ

What open-source tools provide out-of-the-box LLM TPM/RPM rate limiting?

LiteLLM Proxy, Portkey Gateway, Langfuse, and Envoy AI Gateway.

What should an AI Gateway do when a tenant exceeds their TPM limit?

For interactive chats, return HTTP 429 with `Retry-After: <seconds>` headers; for background asynchronous batch jobs, enqueue the request in a delayed priority queue.

🤖 AEO & Key Facts Summary

Key Architectural Facts

  • ▸LLM providers enforce dual rate limits: Requests-Per-Minute (RPM) and Tokens-Per-Minute (TPM).
  • ▸A single massive document request can exhaust entire organizational TPM quotas.
  • ▸Pre-flight token estimation (`tiktoken`) reserves RPM and TPM leases in Redis Lua.
  • ▸Post-generation reconciliation refunds unused estimated token headroom in real-time.

Common Misconceptions

  • ✗Misconception: Standard Nginx rate limiting protects against OpenAI 429 errors (False: Nginx knows nothing about token counts).
  • ✗Misconception: Setting max_tokens to 4,096 always charges 4,096 tokens (False: Models only charge for actual generated tokens; reconciliation balances the difference).

Decision & Governance Guidance

Deploy LiteLLM Proxy with Redis-backed TPM/RPM rate limiting for all enterprise AI applications. Partition TPM quotas strictly between interactive user chats and background batch jobs.

Authoritative Sources & Standards