An LLM API Timeout Strategy That Survives Slow Generations
Why a single fixed timeout breaks LLM calls, and how to set connect, first-token, and idle deadlines, stream to stay alive, and cancel cleanly with AbortController.
A request to a chat model that usually answers in three seconds can occasionally take ninety. Set one blunt 30-second timeout across the client and you will cut off the slow-but-correct generations while doing nothing for the truly stuck ones. The fix is not a bigger number — it is several deadlines that each measure a different thing: how long you wait to connect, how long you wait for the first token, and how long you tolerate silence once tokens are flowing. This article shows how to build that layered timeout, why streaming changes the math, and how to cancel work the instant a deadline or a user says stop.
LLM latency is unusual, and that is the root of the problem. A normal REST call has a latency distribution clustered tightly around a mean, so a fixed timeout a few multiples above the mean is safe. Model generation latency is heavy-tailed and depends on things the client cannot see: the output length the model chooses, whether it is a reasoning model spending hidden tokens before it writes a word, upstream queueing, and cold routing. A timeout tuned to the median amputates the tail; a timeout tuned to the tail lets genuinely hung requests hold connections and quota for far too long.
Separate the deadlines that a timeout is trying to cover
"Timeout" is one word for at least four independent limits. Naming them separately is the whole trick.
- Connect timeout — the ceiling on establishing the TCP/TLS connection. This should be short (a couple of seconds); a slow connect is a network or DNS problem, not a slow model.
- Time-to-first-token (TTFT) deadline — how long you wait for the model to start responding. On a reasoning model this legitimately runs into tens of seconds, because the model may think before emitting anything.
- Idle / inter-token timeout — once a stream starts, the maximum gap you tolerate between chunks. A healthy stream sends tokens continuously; a long silence means the upstream stalled.
- Total / wall-clock deadline — an absolute cap on the whole request so a pathological case cannot run unbounded.
A single fixed timeout collapses all four into one number and therefore serves none of them well. Long-generation reliability comes from setting a generous TTFT and total budget while keeping the idle timeout tight, so you fail fast on a stall but stay patient during real work.
Streaming turns a hard total timeout into a soft idle timeout
The most effective change you can make is to stream. When you send stream: true, the model returns tokens over Server-Sent Events instead of buffering the whole answer, and you stop measuring "did the entire response finish in N seconds" and start measuring "did the next chunk arrive in N seconds." That inverts the problem in your favor: a five-thousand-token answer no longer risks a total-time cutoff, because each chunk resets the idle clock. If you have not built a streaming client yet, streaming LLM responses with SSE covers the wire format and parsing; here we care only about the timing behavior it unlocks.
Idle-timeout logic on a fetch stream looks like this — a watchdog that is reset on every chunk and fires only during real silence:
async function streamWithIdleTimeout(url, body, { idleMs = 20000 } = {}) {
const controller = new AbortController();
let timer = setTimeout(() => controller.abort(new DOMException("idle", "TimeoutError")), idleMs);
const res = await fetch(url, {
method: "POST",
headers: { "Content-Type": "application/json", Authorization: "Bearer AVE_GATEWAY_API_KEY" },
body: JSON.stringify({ ...body, stream: true }),
signal: controller.signal,
});
const reader = res.body.getReader();
const decoder = new TextDecoder();
try {
while (true) {
const { value, done } = await reader.read();
if (done) break;
clearTimeout(timer); // a chunk arrived: reset the watchdog
timer = setTimeout(() => controller.abort(new DOMException("idle", "TimeoutError")), idleMs);
process.stdout.write(decoder.decode(value, { stream: true }));
}
} finally {
clearTimeout(timer);
}
}
Notice the idle budget (20 seconds of silence) is decoupled from total time. The request can run for minutes as long as it keeps producing tokens, but a genuine stall trips within one idle window. That is exactly the behavior a fixed total timeout cannot express.
Cancel with AbortController, and read the error name
Every timeout needs a mechanism to actually stop the work, and in the browser and modern runtimes that mechanism is AbortController. You pass its signal into fetch, and calling abort() tears the connection down. For a plain wall-clock cap without hand-rolling a controller, AbortSignal.timeout(ms) returns a signal that aborts itself after the given milliseconds:
// Hard 60s cap on a non-streaming call:
const res = await fetch(url, { method: "POST", signal: AbortSignal.timeout(60000), /* ... */ });
The two cancel paths report different errors, and distinguishing them is what lets you decide whether to retry. A manual controller.abort() rejects the fetch with a DOMException named AbortError; a signal from AbortSignal.timeout() rejects with one named TimeoutError. To combine "user pressed stop" with "hard deadline," merge signals using AbortSignal.any([userSignal, AbortSignal.timeout(60000)]), then branch on err.name:
try {
await callModel();
} catch (err) {
if (err.name === "TimeoutError") retryOrEscalate(); // our deadline fired
else if (err.name === "AbortError") stopQuietly(); // the user cancelled
else throw err; // a real failure
}
Treating a user cancel as a failure to retry is a classic bug: you re-fire a request the user explicitly killed, pay for the tokens, and pollute your metrics. Branch on the name.
The server side has its own timeouts, and they win
If your client talks to a gateway or your own backend rather than directly to the provider, that hop imposes its own deadlines, and the shortest one along the path decides the outcome. A common failure is a client patiently waiting five minutes while an intermediate proxy severs the connection at sixty seconds, or a load balancer's idle timeout closing a streaming response that "hangs" because response buffering delayed the first byte past the limit. Three server-side rules keep this from biting:
- Propagate cancellation upstream. When the client disconnects, abort the request you made to the provider. Cancellation is best-effort — the provider may keep generating, and briefly billing, for a moment after you disconnect — but propagating the abort is what gives you any chance of stopping the spend on tokens no one will read. This is the server mirror of
AbortController. - Do not buffer streamed responses. A proxy that accumulates the body before forwarding delays the first byte and can trip an upstream idle timeout that would never have fired on a flushed stream.
- Give the deadline to the runtime. In Go, a
context.Contextdeadline is the idiomatic carrier; when it expires, the outbound HTTP request is cancelled and resources are released. Any language has an equivalent — the point is that the timeout must reach the socket, not just the wrapper.
A gateway is a natural place to enforce a consistent policy, because it already terminates client connections and holds provider credentials. Registering each upstream behind it lets one deadline and one cancellation path apply no matter which provider serves the request, and the tokens actually consumed before an abort are recorded in its usage logs so a cancelled call is still accounted for. Bear in mind that the local count is best-effort — when the final upstream usage never arrives it may be estimated — so reconcile against the provider's own usage for anything billing-critical.
Timeouts and retries are one system, not two
A timeout that fires is a signal to your retry logic, so the two must be designed together or they fight. Two rules prevent the common pathologies:
- A retry only makes sense after a fast failure. If your total timeout is long and you retry on expiry, a single slow request becomes several slow requests stacked back to back, multiplying latency and cost. Keep the per-attempt deadline tight enough that a retry is cheaper than continuing to wait, and cap the number of attempts.
- A timed-out request may still have executed. When you cut a connection, the provider might have already produced the response you never read. Retrying blindly can double-charge or duplicate side effects. Pair timeouts with an idempotency strategy so a retry after a timeout is safe — the mechanics are in LLM API retries and idempotency.
Timeouts also interact with rate limits. A burst of timeouts followed by immediate retries looks, from the upstream's perspective, like a load spike, and can push you into 429 responses. Coordinating deadlines with backpressure — queueing rather than hammering — is covered in LLM API rate limits and backpressure. When a whole provider is slow rather than a single request, the right move is often to shift traffic; see LLM API failover strategy.
Reasoning models need a longer patience budget
Reasoning models change the timing profile enough to deserve their own numbers. Because they spend hidden reasoning tokens before producing visible output, TTFT can be long even when the model is working perfectly, and a TTFT deadline calibrated for a fast chat model will kill them prematurely. Two adjustments help: raise the TTFT and total budgets specifically for reasoning-model routes, and prefer streaming so that once visible tokens begin, your idle timeout — not a total cap — governs. Tune reasoning parameters separately, and reserve enough hidden-token headroom using the policy in LLM context window management.
FAQ
What timeout value should I use for an LLM API call?
There is no single value, which is the point of a layered strategy. Use a short connect timeout (a couple of seconds), a generous TTFT deadline sized to your slowest model family, a tight idle timeout for streamed responses (tens of seconds of silence), and an absolute total cap as a backstop. Tuning one number forces you to trade tail correctness against stall detection; separate deadlines let you have both.
How do I tell a deadline timeout apart from a user cancellation?
Check the error's name. A signal from AbortSignal.timeout() rejects with TimeoutError; a manual controller.abort() rejects with AbortError. Retry or escalate on TimeoutError, stop silently on AbortError, and re-throw anything else. Conflating the two causes you to retry requests the user deliberately killed.
Does aborting a request actually stop the provider from billing me?
Only if the cancellation reaches the provider, and even then it is best-effort. In the browser, controller.abort() closes the connection to your server; your server must then abort its own upstream request, or generation is likely to continue and you are billed for tokens no one reads. Propagating cancellation across every hop is what gives an abort a chance to act as real cost control — confirm it worked by cross-checking the provider's usage or invoice, since your local token count can be an estimate.
Why does my streaming response hang until it finishes instead of streaming?
Almost always response buffering in an intermediary. A reverse proxy or framework that accumulates the body before forwarding defeats streaming and can also trip idle timeouts by delaying the first byte. Disable buffering for text/event-stream responses and flush chunks as they arrive.
Split one route into four deadlines
Take one production route and split its single timeout into four: a connect timeout, a TTFT deadline, a stream idle timeout, and a total cap. Switch that route to stream: true, wire the idle watchdog above, then deliberately abort a request mid-stream. Your locally recorded output tokens should fall, which is a good sign the deadline reached the socket — but treat it as a signal rather than proof, and reconcile against the provider's own usage when the cost matters, since the upstream can keep billing briefly after you disconnect.