LLM Context Window Management for Long Conversations
The context window has two costs: overflow errors and per-turn billing. How to count tokens, trim and summarize history, cache stable prefixes, and use server-side compaction.
A growing conversation fails in two different ways, and confusing them leads to the wrong fix. The loud failure is overflow: the prompt plus the requested output exceeds the model's context window and the request is rejected outright. The quiet failure is cost: everything you keep in context is re-billed on every single turn, so a chat that never overflows can still get expensive linearly as history piles up. Managing a context window means defending against both — keeping the token count under the hard limit and keeping it small enough that you are not paying to resend the same history forever. This article covers how to measure the budget, the trimming and summarization strategies that keep it in check, and the caching and server-side compaction that reduce what a long conversation costs.
One budget, shared by input, output, and reasoning
The context window is a single token budget that the whole request draws from. It is not "input tokens up to the limit, then output on top" — input tokens, the tokens you ask the model to generate, and (on reasoning models) the hidden reasoning tokens all compete for the same ceiling. Two consequences follow immediately. First, a long prompt shrinks the room left for the answer, so an over-stuffed context can force a truncated or empty response even without a hard overflow error. Second, on reasoning models the invisible reasoning tokens eat into that ceiling too, which is why reasoning model API parameters stresses reserving output headroom. Budget the window as a whole: decide up front how many tokens you will let history occupy, how many you reserve for reasoning, and how many you guarantee for the visible answer.
By default, exceeding the window is a hard error — with truncation disabled, providers return a context-length error rather than silently dropping tokens — so overflow is usually your problem to prevent, not the model's to absorb. There is one deliberate exception worth knowing: the Responses API accepts truncation: auto, which lets the provider drop items from the middle of the input to make a too-large request fit instead of rejecting it. That trades a clean failure for a silent content loss you did not choose, so use it knowingly and log when it fires; automatic truncation behavior and what it drops vary by provider. Either way, the reliable defense is the same: know your token count before you send.
Measure before you send
You cannot manage a budget you do not measure. Every history-management decision depends on an accurate token count of the request you are about to make, and "roughly four characters per token" is a rule of thumb that drifts badly on code, non-Latin scripts, and structured data. Count with the provider's own tokenizer where you can, and count the whole request — system instructions, tools, every message, and any attached documents — not just the latest user turn.
The counting also has to be honest about what the model sees. Tool definitions, JSON schemas for structured outputs, and images all consume tokens, and they are easy to forget because they do not look like "conversation." A large tool catalog can quietly occupy a meaningful slice of the window on every call. Once you have a reliable count, the strategies below are just policies for keeping that number where you want it.
Trimming strategies, from crude to clever
There are four common ways to keep history under budget, and real systems combine them.
Truncate the oldest turns. The simplest policy: keep the system prompt plus the most recent N turns and drop the rest. It is cheap and predictable, and it is the right default for support-style chats where old context rarely matters. Its weakness is amnesia — the model forgets facts established early in a long session.
Summarize and replace. Instead of dropping old turns, compress them. Periodically ask a model to summarize the earliest part of the conversation into a compact note, then replace those turns with the summary. This preserves the gist at a fraction of the tokens, at the cost of an extra call and some fidelity loss. The trade is usually worth it for long assistants that must remember decisions made an hour ago.
Retrieve instead of retain. For knowledge that lives in documents rather than in the dialogue, do not carry it in context at all. Store it externally, and on each turn retrieve only the passages relevant to the current question and inject those. Retrieval keeps the window small and constant regardless of how much source material exists — the difference between paying for a whole manual every turn and paying for the two paragraphs that matter.
Keep a stable prefix and vary the tail. Structure the conversation so the unchanging part (system prompt, tools, reference material) sits at the front and only the recent turns change at the end. This is good hygiene on its own, and it is the precondition for caching, discussed next.
Most durable chat systems layer these: a fixed system prefix, a running summary of the middle, the last few verbatim turns, and retrieval for anything factual. The exact mix depends on whether your priority is recall, latency, or cost.
Cache the prefix you cannot avoid resending
Some context genuinely must be present every turn — a detailed system prompt, a fixed toolset, a document under discussion. You cannot trim it without breaking the task, but you can stop paying full price to reprocess it, and that is exactly what prompt caching does. When the stable prefix is cached, the provider reuses it and bills those tokens at a steep discount instead of the full input rate. The management move is to arrange the conversation so the cacheable region is maximal and byte-stable — no timestamps or reordered tools inside it — and let the volatile tail fall after the cache boundary. The provider-by-provider mechanics are in prompt caching across OpenAI, Claude, and Gemini; the point here is that caching and context management are the same discipline viewed from two angles. Trimming reduces how many tokens you send; caching reduces what the unavoidable tokens cost.
Server-side state does not make history free
The Responses API lets you continue a thread with previous_response_id instead of resending the whole history, which is a real ergonomic win — you send only the new turn and the server stitches it to the stored context. It is tempting to read this as "the earlier context is now free." It is not. All previous input tokens in the chain are still billed as input tokens on each continuation; chaining saves you from sending bytes over the wire, not from paying for the context the model still processes. So server-side conversation state changes who holds the history, not whether you pay for it. You still need the trimming and caching above to control cost; previous_response_id is orthogonal to that.
Compaction: shrinking a conversation on the server
When a conversation grows long enough that even summarization on your side is awkward, some Responses-style workflows expose a server-side compaction step that condenses the running context into a smaller representation you can continue from. This gateway surfaces that as a dedicated endpoint, POST /v1/responses/compact, which relays a compaction request to an upstream that supports it. The request carries the conversation to condense — model, input, instructions, and optionally previous_response_id, tools, and a reasoning object — and the response returns a compacted output plus a usage object with the input and output tokens the compaction itself consumed.
curl https://your-gateway.example.com/v1/responses/compact \
-H "Authorization: Bearer AVE_GATEWAY_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"model": "gpt-5.6",
"previous_response_id": "resp_abc123",
"instructions": "Condense the conversation so far, preserving decisions and open tasks."
}'
Two things to keep in mind. Compaction is itself a model call — it reads the conversation and writes a condensed version, so it consumes tokens that show up in your usage logs like any other request; it trades a one-time compaction cost for cheaper subsequent turns. And it is only meaningful against an upstream that implements the compaction endpoint, so it is a passthrough capability, not something the gateway can synthesize for a provider that lacks it. Where it is available, it is the server-side analogue of the "summarize and replace" strategy, moving the compression work upstream instead of running it in your own client.
Reserve room for the answer
Whatever mix you choose, always leave the visible answer enough space. Set an explicit output cap and make sure context_used + reserved_output stays comfortably under the window, remembering that on reasoning models the reserve must also cover hidden reasoning tokens. Skip this and you get the frustrating failure where a request succeeds, bills you for a large input, and returns a truncated answer because there was no room left to finish. Apply the same principle to timeouts: leave slack on purpose.
Bringing it together
A context policy for a long-running assistant usually looks like this: pin a cacheable system prefix, keep the last handful of turns verbatim, maintain a rolling summary (or a server-side compaction) of the older middle, retrieve facts from an external store rather than carrying them, count tokens before every send, and reserve a fixed output budget. Each piece targets a specific part of the two-cost problem — overflow or per-turn billing — and together they keep a conversation both under the hard limit and off an ever-rising cost curve. The gateway's role is to make the effect visible: one place to watch token counts per request and confirm that a change to your history policy actually moved the number, which ties into the broader work of reducing LLM API costs.
FAQ
What happens when a request exceeds the context window?
By default the provider rejects it with a context-length error; it does not silently truncate your prompt to fit, which makes overflow your responsibility to prevent by counting tokens before sending. The exception is Responses' opt-in truncation: auto, which drops middle input items to squeeze an oversized request in — convenient, but a silent content loss, so enable it deliberately rather than as a safety net. Because input, output, and reasoning tokens share one budget, an over-long prompt can also cause a truncated answer even below the hard limit, since there is too little room left to generate.
Does using previous_response_id reduce my token bill?
No. It saves you from resending history over the wire, but all previous input tokens in the chain are still billed as input on each continuation. Server-side state is an ergonomics and latency feature, not a cost reduction. To actually cut context cost you still need trimming, summarization, and prompt caching of the stable prefix.
Summarize history or truncate it — which should I use?
Truncation (keep the last N turns) is simplest and fine when old context rarely matters, but it forgets early facts. Summarization compresses old turns into a compact note, preserving the gist at extra cost per compression. Long assistants that must remember earlier decisions usually run both: verbatim recent turns plus a rolling summary of the older middle, with retrieval for anything factual.
What does the /v1/responses/compact endpoint do?
It relays a compaction request to an upstream that supports condensing a Responses conversation into a smaller representation you can continue from. You pass the conversation (via input/instructions and optionally previous_response_id), and it returns a compacted output and a usage count for the tokens compaction consumed. It is a passthrough capability — available only when the upstream implements it — and it is the server-side version of summarize-and-replace.
Plot the curve, then flatten it
Instrument one long-running conversation to log its total token count on every turn, then plot the curve. If it rises steadily, add a rolling summary or a compaction step at a threshold and watch the curve flatten in your usage logs. Then mark your system prefix as cacheable and confirm the per-turn input charge drops. Those two before/after curves — token count and input cost — tell you whether your context policy is actually working, on your own traffic rather than in theory.