Skip to main content

> multi-tier_llm_rate_limiting:_tokens-per-minute_(tpm)_&_requests-per-minute_(rpm)

Multi-Tier LLM Rate Limiting: Tokens-Per-Minute (TPM) & Requests-Per-Minute (RPM)

Why do standard API rate limiters fail to prevent upstream LLM provider 429 errors, and how does dual-dimensional TPM/RPM sliding window scheduling smooth AI traffic bursts?

Senior (L5)

⚡THE SHORT ANSWER

Standard API rate limiters (e.g. Nginx or Cloudflare) measure traffic strictly in Requests Per Minute (RPM): treating every incoming HTTP request as an equal unit of load. In Generative AI systems, RPM rate limiting is completely insufficient because LLM providers (OpenAI, Anthropic, Google) enforce strict dual quotas: Requests Per Minute (RPM) AND Tokens Per Minute (TPM). A single user sending a 100,000-token PDF analysis request consumes the same RPM as a 20-token greeting, but consumes 5,000 imes more TPM quota, instantly triggering upstream HTTP 429 Rate Limit rejections for all other users. Production AI gateways (LiteLLM, Portkey, Envoy AI Gateway) implement Multi-Tier Dual-Dimensional Token Bucket Scheduling:

1

Fast Pre-Flight Token Estimation (counting input prompt tokens via tiktoken),

2

Atomic Redis Token & Request Leases (reserving both estimated TPM and 1 RPM upfront), and

3

Post-Generation Reconciliation (refunding or settling the exact token delta after streaming finishes).

Engineering Handbook & Failure Dynamics

6-Dimensional Architecture Breakdown

⚙️1. Underlying Mechanism

Execution

Dual-dimensional LLM rate limiting executes across four coordinated steps:

1

Prompt Token Sizing: Incoming prompt text is tokenized in-memory via SIMD tokenizer (e.g. tiktoken in <0.2ms), estimating total tokens T_{ ext{est}} = T_{ ext{prompt}} + ext{max_output_tokens}.

2

Atomic Double Lease (Redis Lua): Atomically checks and decrements both RPM bucket (R ge 1) and TPM bucket (T ge T_{ ext{est}}). If insufficient TPM, the request is queued in an in-memory priority delay queue rather than rejected.

3

Upstream LLM Execution: The request is forwarded to OpenAI/Anthropic.

4

Reconciliation: Upon stream completion, exact actual tokens T_{ ext{actual}} are calculated from the response metadata, and the difference T_{ ext{est}} - T_{ ext{actual}} is immediately refunded back to the tenant's TPM bucket.

🎯2. Appropriate Use Context

Scope

Multi-tenant SaaS LLM gateways, internal developer AI proxy platforms, automated batch document ingestion pipelines, and enterprise AI portals.

⚠️3. Production Failure Modes

P0 Risk
  • ✓

    A single tenant running 50 concurrent automated document summarization threads that exhaust the organization's entire 1,000,000 TPM OpenAI tier-4 limit, causing 100% of executive and customer-facing chatbot queries to fail with HTTP 429 for 3 minutes

  • ✓

    static TPM limits that fail to dynamically back off when upstream providers throttle under peak internet congestion

📡4. Diagnostic Signals & Telemetry

Telemetry
  • ✓

    Upstream provider error logs showing RateLimitError: Rate limit reached for default-gpt-4o in organization on tokens_per_min

  • ✓

    API gateway metrics showing 0% RPM utilization but 100% TPM exhaustion

  • ✓

    client requests timing out during batch workload spikes

🛡️5. Prevention & Safeguards

Safeguards
  • ✓

    Deploy an AI gateway proxy (LiteLLM / Envoy) with dual TPM/RPM sliding window limiters

  • ✓

    partition TPM quotas strictly across tenants/services (customer_support: 60%, batch_jobs: 20%, internal_dev: 20%)

  • ✓

    implement automated provider failover routing (e.g. spilling over from OpenAI to Azure OpenAI or Anthropic Claude when TPM exceeds 85%)

⚖️6. Architectural Trade-offs

Trade-off

Token pre-flight counting and reconciliation adds ~1-2ms of gateway overhead and requires Redis infrastructure, but completely eliminates cascading upstream HTTP 429 outages.

📋

Case Study (TinyCTO In-Field Example)

REAL-WORLD TELEMETRY

A SaaS company with 10,000 active users suffered frequent OpenAI outages during morning peaks because marketing teams launched automated batch copy-generation jobs that burned 800,000 TPM in 10 seconds. The infrastructure team deployed LiteLLM with Dual TPM/RPM Token Buckets and assigned tenant tiers: Marketing Batch was capped at 150,000 TPM with asynchronous delay queues, while Live Customer Chat was guaranteed 600,000 TPM with priority routing. Production HTTP 429 errors dropped from 1,200/day to exactly 0.

Interactive Concept Drills

2 Cards
Q1

Why is traditional Requests-Per-Minute (RPM) rate limiting insufficient for Generative AI APIs?

Because requests have wildly differing payload sizes: a single large document prompt can consume 100,000 Tokens-Per-Minute (TPM), exhausting provider token quotas even when RPM is well below the limit.
Q2

How does Token Reconciliation work in dual-dimensional rate limiters?

The gateway reserves an estimated token count before sending the request; once generation streams to completion, unused reserved tokens are refunded back to the user's quota bucket in Redis.

Multi-Tier LLM Rate Limiting: Tokens-Per-Minute (TPM) & Requests-Per-Minute (RPM) — Technical FAQ

What open-source tools provide out-of-the-box LLM TPM/RPM rate limiting?

LiteLLM Proxy, Portkey Gateway, Langfuse, and Envoy AI Gateway.

What should an AI Gateway do when a tenant exceeds their TPM limit?

For interactive chats, return HTTP 429 with `Retry-After: <seconds>` headers; for background asynchronous batch jobs, enqueue the request in a delayed priority queue.

🤖 AEO & Key Facts Summary

Key Architectural Facts

  • ▸

    LLM providers enforce dual rate limits: Requests-Per-Minute (RPM) and Tokens-Per-Minute (TPM).

  • ▸

    A single massive document request can exhaust entire organizational TPM quotas.

  • ▸

    Pre-flight token estimation (tiktoken) reserves RPM and TPM leases in Redis Lua.

  • ▸

    Post-generation reconciliation refunds unused estimated token headroom in real-time.

Common Misconceptions

  • ✗

    Misconception: Standard Nginx rate limiting protects against OpenAI 429 errors (False: Nginx knows nothing about token counts).

  • ✗

    Misconception: Setting max_tokens to 4,096 always charges 4,096 tokens (False: Models only charge for actual generated tokens; reconciliation balances the difference).

Decision & Governance Guidance

Deploy LiteLLM Proxy with Redis-backed TPM/RPM rate limiting for all enterprise AI applications. Partition TPM quotas strictly between interactive user chats and background batch jobs.

Authoritative Sources & Standards

Technical terms on this page

agent instructionProduction