LLM API Rate Limits and Backpressure That Holds

When an LLM provider rate-limits you, it returns HTTP 429, and the naive reaction — retry immediately — makes things worse, because unsuccessful requests still count against your per-minute limit. The durable fix is two layers working together: read the rate-limit headers the provider sends so you know how much budget remains, and apply backpressure on your side — bounded queues, concurrency caps, and backoff with jitter — so that when demand exceeds supply, your system slows down instead of melting down. This article covers both layers and the code to implement them.

Rate limits are not an error condition to be papered over; they are a capacity signal. A system that treats them as signal degrades predictably. A system that ignores them oscillates between hammering the provider and dropping user requests.

Read the headers before you tune anything

OpenAI returns rate-limit state on every response through headers you can inspect directly. The request-based set is:

• x-ratelimit-limit-requests — the ceiling of requests in the window
• x-ratelimit-remaining-requests — how many you have left
• x-ratelimit-reset-requests — time until the request window resets

There is a parallel token-based set — x-ratelimit-limit-tokens, x-ratelimit-remaining-tokens, x-ratelimit-reset-tokens — and project-scoped variants such as x-ratelimit-remaining-project-tokens. Most real limits bite on tokens, not request count, because a single long-context call can consume a large slice of your token budget. Log these headers on every response and you can see throttling coming before it becomes a wall of 429s:

resp = client.chat.completions.with_raw_response.create(
    model="gpt-5.6",
    messages=[{"role": "user", "content": prompt}],
)
h = resp.headers
print(h.get("x-ratelimit-remaining-requests"), h.get("x-ratelimit-remaining-tokens"))

Logging these headers is the operational foundation: once you record x-ratelimit-remaining-requests and x-ratelimit-remaining-tokens per response, you can watch which upstream is approaching its ceiling before the 429s start. A gateway is a good place to centralize that logging, since every response already passes through it — but a per-channel remaining-budget curve is something you build on top of the raw headers, not a figure the response hands you pre-computed.

Retry 429s with exponential backoff and jitter

When a 429 does arrive, back off before retrying. Exponential backoff multiplies the wait after each failure; jitter randomizes it so a fleet of clients does not all retry on the same tick and re-collide — a thundering herd that recreates the overload you are recovering from.

import random, time

def backoff_sleep(attempt, base=0.5, cap=20.0):
    # full jitter: random point in [0, min(cap, base * 2**attempt))
    ceiling = min(cap, base * (2 ** attempt))
    time.sleep(random.uniform(0, ceiling))

for attempt in range(6):
    r = call_model(prompt)
    if r.status_code != 429:
        break
    backoff_sleep(attempt)

Two rules keep this honest. First, cap both the delay and the attempt count — infinite retries just move the outage downstream. Second, only retry what is retryable: 429 and transient 5xx/timeouts are worth retrying; a 400 for a malformed request will fail identically every time. Because a retried write can duplicate work, pair retries with idempotency — the full treatment is in LLM API retries and idempotency.

If a response includes a Retry-After header, prefer it over your computed backoff — the server is telling you exactly when the window reopens, and honoring that value both recovers faster and avoids wasting attempts that are guaranteed to fail. Treat your exponential schedule as the fallback for when the server does not supply an explicit hint.

Backpressure: slow the source, do not drop the floor

Retries handle a single failed call. Backpressure handles the situation where your intake rate simply exceeds what the provider will accept. The core idea from queueing theory: put a bounded buffer between producers and the provider, and when the buffer is full, make producers wait rather than accepting unbounded work you cannot serve.

Three mechanisms compose into a backpressure strategy:

Mechanism	What it bounds	Failure mode it prevents
Concurrency limit (semaphore)	In-flight requests	Opening thousands of sockets that all 429
Bounded queue	Work waiting to start	Unbounded memory growth under a spike
Admission control / load shedding	New work accepted	Accepting requests you can never serve in time

A concurrency limit is the highest-leverage change and the easiest to add. Cap simultaneous in-flight calls so you match, rather than exceed, the provider's tolerance:

import asyncio

sem = asyncio.Semaphore(20)  # never more than 20 concurrent upstream calls

async def guarded_call(prompt):
    async with sem:
        return await call_model_async(prompt)

When the semaphore is full, new callers wait — that waiting is backpressure. It propagates the constraint back to the source instead of letting it explode at the provider boundary.

Shed load deliberately when the queue is full

A bounded queue eventually fills during a sustained spike. What happens then is a product decision, and making it explicit is the whole point:

• Reject fast — return a 503 with a Retry-After so callers retry later. Best for interactive traffic where a slow answer is worse than a clear "try again."
• Shed by priority — drop or defer low-priority work (batch, background) so interactive requests keep flowing.
• Degrade — route overflow to a cheaper, faster model that is more likely to have headroom.

The anti-pattern is an unbounded queue that accepts everything and quietly grows latency until requests time out anyway. That converts a visible, recoverable rejection into an invisible, system-wide stall. Bound the queue, and decide what falling off the end means.

Where a gateway centralizes the policy

Backpressure implemented in every service drifts — each one picks different limits and they interact badly. A gateway is the natural place to enforce one coherent policy, because all traffic already passes through it. In this project, provider connections are registered as channels, each with its own quota and priority, and the gateway already fails over to a healthy channel when one starts throttling (LLM API failover strategy) and records per-request usage you can tune from. The richer admission machinery is architecture you layer on at that same choke point:

• track remaining token and request budget per channel by persisting the response headers,
• apply per-channel concurrency limits so one hot channel does not starve others,
• keep one aggregate retry budget so recovery traffic cannot amplify an outage.

Treat those three as a design to implement at the gateway rather than switches that ship on by default; the value of the gateway is that there is a single place to add them. Centralizing matters for a concrete reason: if ten services each retry independently on 429, their combined retry traffic can exceed your limit on its own, and a single shared retry budget is what prevents that self-inflicted amplification.

A layered defense, in order of impact

Apply these in order; each one buys time for the next:

1. Measure — log rate-limit headers so throttling is visible early.
2. Limit concurrency — a semaphore that matches provider tolerance.
3. Bound the queue — a fixed buffer, never unbounded.
4. Back off with jitter — on 429/5xx, never on 4xx client errors.
5. Shed or degrade — an explicit policy for a full queue.
6. Spread load — multiple channels and failover to raise the effective ceiling.

The first two changes usually eliminate most 429 pain. The rest turn the remaining spikes into graceful slowdowns instead of outages.

Budget tokens, not just requests

Because token limits usually bind before request limits, estimate the token cost of a call before you send it and reserve budget accordingly. A request consumes both input tokens (your prompt, plus any chained history) and output tokens (bounded by max_tokens/max_output_tokens). If you admit work based only on request count, a burst of long-context calls can blow through the token ceiling while your request counter looks healthy — and every one of those calls comes back 429.

A practical admission rule is a token-aware gate: track remaining token budget from x-ratelimit-remaining-tokens, estimate each pending call's worst-case tokens, and only admit calls whose estimate fits the remaining headroom. This is the token-bucket idea applied to LLM traffic — refill at the reset interval, spend on admission, and make callers wait when the bucket is low.

• Cap max_tokens/max_output_tokens to the size you actually expect; an oversized cap reserves budget you never use and starves other calls.
• Count chained history: with stored conversation state, earlier input tokens still bill as input tokens, so a long thread costs more per turn than it looks.
• Prefer many small calls to one giant call when a single request would reserve a large slice of the token window at once.

Right-sizing the token estimate is often a bigger lever than tuning concurrency, because it prevents the calls that would 429 from ever being admitted. Cost and capacity are the same dial here, which is why this connects to reducing LLM API costs.

FAQ

What status code means I am being rate-limited, and should I retry it?

HTTP 429. It is retryable, but only with backoff and jitter, never immediately — unsuccessful requests still count toward your per-minute limit, so tight retry loops deepen the throttling. Cap attempts and delay.

Are request limits or token limits the ones I will actually hit?

Usually token limits. Check x-ratelimit-remaining-tokens alongside x-ratelimit-remaining-requests. A workload with long prompts or large max_tokens can exhaust the token budget while barely touching the request count, so budgeting on request count alone misleads you.

Does raising my concurrency limit increase throughput?

Only up to the provider's tolerance. Past that point, extra concurrency produces more 429s, more retries, and lower useful throughput. Tune the semaphore to the largest value where the 429 rate stays near zero, then stop.

Is a bigger queue a fix for sustained overload?

No. A larger queue only delays the moment of truth and inflates tail latency while it fills. If intake exceeds capacity for a sustained period, no buffer saves you — you need load shedding, a higher rate limit, or more channels. Queues absorb bursts, not trends.

Start with one semaphore and header logging

Add one concurrency semaphore around your model calls and start logging x-ratelimit-remaining-tokens on every response. Chart those recorded values against your 429s for a day: if the remaining-token dips line up with the errors, lower the semaphore until the dips stop. That single feedback loop removes most rate-limit incidents before you build anything more elaborate.

References

• OpenAI: Rate limits guide
• OpenAI: Production best practices
• MDN: HTTP 429 Too Many Requests
• MDN: Retry-After header