LLM API Retries and Idempotency Done Safely

Retrying a failed LLM API call is safe only when you can guarantee the call runs at most once even if the network makes it look like it failed twice. The danger is subtle: a request can succeed at the provider, then the response gets lost on the way back, your client sees a timeout, retries, and now you have been charged twice or triggered a side effect twice. The fix is a pair of disciplines that always travel together — retry only the errors worth retrying, with backoff and jitter, and make each retryable request idempotent so a duplicate is recognized and collapsed instead of executed again. This article shows how to do both.

The reason this matters for LLM traffic specifically: generation calls cost real money per token, and agent workflows chain calls where a duplicated step corrupts downstream state. "Just retry on failure" is fine for a GET; it is a billing incident waiting to happen for a POST that generates tokens.

Classify errors before you retry anything

Not every failure should be retried. Retrying a permanent error wastes time and, under rate limiting, actively harms you because unsuccessful requests still count toward your limit. Sort failures into three buckets:

Class	Examples	Action
Retryable transient	timeouts, connection resets, 500, 502, 503, 429	Retry with backoff
Non-retryable client error	400 bad request, 401/403 auth, 404, 422	Fail fast, surface the error
Ambiguous	timeout after the request was sent	Retry only if idempotent

The third row is the dangerous one. A timeout does not tell you whether the provider processed the request — it only tells you the response did not arrive. Retrying an ambiguous failure without idempotency is exactly how double charges happen. That is why the two topics are inseparable.

Retry loop: backoff, jitter, and a cap

A correct retry loop backs off exponentially, adds jitter so concurrent clients do not synchronize, caps the number of attempts, and retries only the retryable classes above.

import random, time
import httpx

RETRYABLE = {429, 500, 502, 503, 504}

def call_with_retries(payload, key, max_attempts=5):
    for attempt in range(max_attempts):
        try:
            r = httpx.post(
                "https://your-gateway.example.com/v1/chat/completions",
                json=payload,
                headers={"Authorization": "Bearer AVE_GATEWAY_API_KEY",
                         "Idempotency-Key": key},
                timeout=30,
            )
        except (httpx.TimeoutException, httpx.TransportError):
            _sleep(attempt); continue          # ambiguous: safe because key is stable
        if r.status_code in RETRYABLE:
            _sleep(attempt); continue
        return r                                # success or non-retryable client error
    raise RuntimeError("exhausted retries")

def _sleep(attempt, base=0.5, cap=15.0):
    time.sleep(random.uniform(0, min(cap, base * (2 ** attempt))))

Notice the Idempotency-Key is passed in and stays constant across every attempt of the same logical request. That single detail is what makes the ambiguous-timeout retry safe. If you generated a fresh key per attempt, you would defeat the entire mechanism.

What an idempotency key actually does

An idempotency key is a unique client-generated identifier — a v4 UUID is the common choice — that you attach to a state-changing request via the Idempotency-Key header. The server stores the outcome of the first request under that key and, on any duplicate carrying the same key, returns the stored result instead of performing the operation again. Generate the key once per logical operation, and reuse it on every retry of that operation:

import uuid
key = str(uuid.uuid4())          # once, before the first attempt
result = call_with_retries(payload, key)

Robust implementations add a fingerprint: a hash of the request body stored alongside the key. If a later request arrives with the same key but a different body, the server rejects it — commonly with HTTP 409 Conflict (an idempotency_conflict) — rather than silently returning the wrong cached result. Keys carry a TTL (24 hours is a typical window); after expiry the key is forgotten and a reuse starts a fresh operation.

A note on provider support: OpenAI documents Idempotency-Key explicitly for its Agentic Commerce checkout flow, where repeating a create or complete call with the same key must return the same result and a parameter mismatch returns 409. General chat and responses endpoints do not advertise it as a headline feature, and most gateways — this project included — do not deduplicate /v1/chat/completions or /v1/responses calls for you today. That gap is exactly why the gateway is the right place to add idempotency: it is the one choke point where you can build the dedup store once and cover every service, regardless of what each upstream does.

Design idempotency at the gateway

When many services call models through one gateway, the gateway is where a duplicate can be caught for the whole system — if you implement the keyed cache. This is the pattern to build, not a switch to flip:

1. Client sends a request with Idempotency-Key: <uuid>.
2. Gateway looks up the key in a short-TTL store (Redis works well).
3. Miss → forward upstream, then store {key → response, fingerprint} before returning.
4. Hit, same fingerprint → return the stored response without calling upstream.
5. Hit, different fingerprint → return 409 Conflict.
6. Hit, still in-flight → return 409 (or wait) so a fast double-submit does not double-execute.

Building this once turns "retry safety" from something every service reimplements into a single enforced property. It also protects against duplication you did not cause — a user double-clicking, a queue redelivering a message, a proxy retrying under the hood. Log the key and its outcome so a suppressed duplicate is auditable in your own store rather than invisible. Retry storms across many channels are another reason centralizing helps: read LLM API rate limits and backpressure for the concurrency side of the same problem.

One honest caveat on guarantees: true at-most-once holds only when the layer that produces the side effect cooperates. If the upstream itself is not idempotent, the gateway's dedup collapses client-visible duplicates but cannot un-send a request the upstream already processed on a lost response. Treat the keyed cache as strong protection against re-submission, and reserve "exactly once" for paths where an upstream idempotency key or atomic side-effect coordination actually exists.

Idempotency and streaming

Streaming complicates dedup because the response is delivered incrementally and may be interrupted mid-flight. If a stream drops after emitting half its tokens, a blind retry with the same key against a naive cache could replay a partial result. Safer patterns: only cache the completed result under the key, treat an interrupted stream as no-result (so the retry regenerates cleanly), or reconnect using the stream's own resumption mechanism rather than re-issuing the request. The SSE resumption details are in streaming LLM responses with SSE.

Retry and idempotency checklist

• Errors classified: retry transient/429/5xx, fail fast on 4xx.
• Ambiguous timeouts retried only when the request is idempotent.
• Backoff is exponential with full jitter and a hard attempt cap.
• One Idempotency-Key per logical operation, reused across all retries.
• Server or gateway stores key + fingerprint with a TTL.
• Duplicate with a mismatched body returns 409, not a stale success.
• In-flight duplicates handled, not double-executed.
• Streaming retries reconnect or regenerate rather than replaying partials.

Choose a retry budget that fails safe

Retries trade latency for success probability, and unbounded retries trade away the whole system. Set an explicit budget with three numbers: maximum attempts, maximum total time, and a per-attempt timeout. When the budget is exhausted, surface a clean error rather than retrying forever — a request that has failed five times with backoff is almost never going to succeed on the sixth, and the attempts are consuming capacity other requests need.

A subtle failure mode appears at scale: if every one of ten services retries independently, their combined retry traffic can exceed your rate limit on its own, which produces more 429s, which trigger more retries. This is a retry storm, and it is self-inflicted. The primary guard is a retry budget as a fraction of total traffic — cap retries at, say, 10% of requests so a broad outage cannot double your load. When a whole channel is failing rather than individual calls, the right response is to stop retrying into a wall and shift traffic; that circuit-breaker-and-failover machinery is owned by LLM API failover strategy, so pair this budget with it rather than duplicating the breaker logic here.

• Bound attempts and total wall-clock time, not just attempt count.
• Cap aggregate retries so recovery traffic cannot amplify an outage.
• Hand sustained per-channel failure to failover instead of retrying blindly.

Observability closes the loop: instrument your own retry rate — the fraction of calls that are retries — so a rising number warns you before it becomes an incident. That metric is one you compute from your call logs; a rising retry fraction is the earliest signal that a budget needs tightening.

FAQ

If a request times out, was it processed or not?

You cannot tell from a timeout alone — it only means the response did not arrive, not that the work did not happen. Treat it as ambiguous. Retry it only if the request carries a stable idempotency key so a duplicate execution is collapsed server-side; otherwise surface the ambiguity to the caller.

Should I generate a new idempotency key for each retry?

No — that is the classic mistake. Generate one key per logical operation and send the identical key on every retry of it. A new key per attempt makes each retry look like a brand-new request, so duplicates execute normally and you lose all protection.

What if two identical requests arrive at almost the same instant?

Handle the in-flight case explicitly. Before the first request finishes, the second one with the same key should get a 409 Conflict (or block until the first completes) rather than being forwarded — otherwise a fast double-submit races past the cache and executes twice.

Prove the dedup on a real duplicate

Wrap your model calls in a retry helper that takes an Idempotency-Key as an argument and generate that key once per operation. Then stand up the keyed store described above and send a deliberate duplicate through it: confirm in your own dedup log that the second call was suppressed and billed once. Until that store exists, a repeated key is just a header the upstream ignores — proving suppression on a real duplicate is the only test that counts.

References

• OpenAI: Production best practices (retries and backoff)
• OpenAI: Agentic Commerce — production and idempotency
• MDN: Idempotency-Key header
• MDN: Idempotent request methods