Streaming LLM Responses with Server-Sent Events
How LLM token streaming works over SSE: the text/event-stream format, parsing data lines and the [DONE] sentinel, why fetch beats EventSource, and cancelling with AbortController.
LLM APIs stream token-by-token output using Server-Sent Events (SSE), a one-way text protocol where the server sends text/event-stream data and the client reads it as it arrives. When you set stream: true, the response is not one JSON body — it is a sequence of small events, each a data: line carrying a JSON fragment, separated by blank lines, and (for Chat Completions) ending with a literal data: [DONE]. To render tokens as they generate, you parse that stream incrementally rather than awaiting a whole response. This article explains the wire format, why you should use fetch instead of the browser's EventSource for LLM calls, and how to cancel a stream cleanly with AbortController.
Streaming is worth the extra parsing because it collapses perceived latency. Time-to-first-token is what a user feels; a streamed answer starts appearing in a few hundred milliseconds instead of after the full generation. The trade is that your client now owns a parser and a cancellation path.
The SSE wire format, precisely
SSE is defined by a small set of rules. The server responds with Content-Type: text/event-stream, the body is UTF-8 text, and messages are separated by a pair of newlines (\n\n). Each message is one or more field: value lines. Only four fields are defined:
| Field | Meaning |
|---|---|
data | The payload. Consecutive data: lines in one message are concatenated with a newline between them. |
event | A name for the event type; dispatched to a matching listener. If omitted, it is a default message. |
id | Sets the client's "last event ID," echoed back on reconnect via the Last-Event-ID header. |
retry | Reconnection delay in milliseconds. |
A line beginning with a colon is a comment and is ignored — providers use these as keep-alive pings to stop idle proxies from closing the connection. A raw Chat Completions stream looks like this on the wire:
data: {"id":"chatcmpl-x","choices":[{"delta":{"role":"assistant"}}]}
data: {"id":"chatcmpl-x","choices":[{"delta":{"content":"Hello"}}]}
data: {"id":"chatcmpl-x","choices":[{"delta":{"content":" world"}}]}
data: [DONE]
Each chunk is a chat.completion.chunk object; the text lives in choices[0].delta.content, and you accumulate deltas to build the message. The stream ends with the sentinel data: [DONE], which is not JSON — you must check for it as a literal string before attempting to parse, or your JSON decoder will throw on the last line.
Two event vocabularies: Chat Completions vs Responses
The transport is identical, but the events differ by endpoint. Chat Completions emits chat.completion.chunk deltas and the [DONE] sentinel. The Responses API emits semantic events with explicit names — response.created, response.output_text.delta, response.completed, and others — where each event announces what it is. Semantic events are easier to route: instead of inspecting the shape of a chunk, you switch on the event name. If you are choosing between the endpoints, see Responses API vs Chat Completions; to move existing streaming code, see migrate Chat Completions to the Responses API.
To get a final usage tally on a Chat Completions stream, send stream_options: {"include_usage": true}; the provider appends a final chunk whose usage object holds the token counts, which you can forward to your gateway's usage logs for accounting.
Why not EventSource?
The browser ships a built-in SSE client, EventSource, and it is the wrong tool for LLM calls for two concrete reasons. First, EventSource only issues GET requests, but LLM calls are POST with a JSON body. Second, EventSource cannot set custom headers, so you cannot send Authorization: Bearer .... Both limits are structural, not configuration you can flip.
The correct approach is fetch with a readable stream, parsing SSE yourself. This gives you POST, headers, and full control over cancellation:
const res = await fetch("https://your-gateway.example.com/v1/chat/completions", {
method: "POST",
headers: {
"Content-Type": "application/json",
Authorization: "Bearer AVE_GATEWAY_API_KEY",
},
body: JSON.stringify({ model: "gpt-5.6", stream: true,
messages: [{ role: "user", content: "Stream a haiku about latency." }] }),
});
const reader = res.body.getReader();
const decoder = new TextDecoder();
let buffer = "";
while (true) {
const { value, done } = await reader.read();
if (done) break;
// Normalize CRLF so the blank-line separator matches regardless of line endings.
buffer += decoder.decode(value, { stream: true }).replace(/\r\n/g, "\n");
const events = buffer.split("\n\n"); // messages end at a blank line
buffer = events.pop(); // keep the incomplete trailing chunk
for (const evt of events) {
// One event can hold several data: lines; join them per the SSE spec.
const data = evt
.split("\n")
.filter((l) => l.startsWith("data:"))
.map((l) => l.slice(5).replace(/^ /, ""))
.join("\n");
if (!data) continue;
if (data === "[DONE]") return; // sentinel is not JSON
const json = JSON.parse(data);
const token = json.choices?.[0]?.delta?.content;
if (token) appendToken(token); // browser: write to a DOM text node
}
}
appendToken is your render sink — in the browser, append the token to a text node or React state; in Node you would write it to process.stdout instead. Two parsing details are load-bearing and easy to skip. Normalize \r\n to \n first, because some intermediaries emit CRLF and a naive split on \n\n then misses the \r\n\r\n boundary. And join all data: lines in an event rather than reading only the first: the SSE spec concatenates consecutive data: lines with a newline, so a multi-line payload is corrupted if you keep just one line.
The buffering matters: TCP does not respect message boundaries, so a single read() can deliver half an event or several events at once. Split on the blank-line separator and carry the incomplete tail forward to the next read. Skipping this is the most common streaming bug — it works in testing and corrupts under load.
Cancelling a stream with AbortController
Users abandon requests: they navigate away, edit the prompt, or stop a runaway generation. You should stop billing tokens the moment they do. AbortController is the mechanism. Create a controller, pass its signal to fetch, and call abort() to tear down the connection:
const controller = new AbortController();
const res = await fetch(url, { method: "POST", signal: controller.signal, /* ... */ });
// stopButton.onclick:
controller.abort(); // fetch promise rejects with a DOMException named "AbortError"
When you call abort(), the in-flight fetch rejects with a DOMException whose name is AbortError, so distinguish a deliberate cancel from a real failure by checking err.name === "AbortError". For a hard timeout without wiring your own controller, AbortSignal.timeout(ms) returns a signal that aborts automatically after the given milliseconds — a firing timeout rejects with a TimeoutError rather than an AbortError. Aborting promptly is a direct cost control, which connects to reducing LLM API costs.
Reconnection and its limits for LLM streams
SSE has built-in reconnection: the id field and retry field let a browser resume after a dropped connection by resending the Last-Event-ID. This is powerful for event feeds but only partially applicable to LLM generation, because a resumed connection cannot un-bill tokens already produced, and most model streams are not designed to replay from an arbitrary offset. For LLM traffic, treat an interrupted stream as an incomplete result and regenerate — coordinating that with idempotency so you are not charged twice, as covered in LLM API retries and idempotency. One infrastructure note: SSE over HTTP/1.1 is capped at six concurrent connections per domain, so streaming many parallel generations from one browser origin needs HTTP/2, which a gateway typically terminates for you.
Proxying an SSE stream through a server
Most production setups do not stream directly from the browser to the provider — they stream through their own backend or gateway, which holds the provider key and applies auth, rate limits, and logging. Proxying SSE has a few requirements that are easy to get wrong and produce a stream that "hangs" until the whole response is ready.
- Disable response buffering. Reverse proxies and frameworks often buffer responses by default, which defeats streaming entirely. Send the chunks through immediately and, where relevant, disable proxy buffering so nothing accumulates the body before forwarding it.
- Set the right headers. Respond with
Content-Type: text/event-streamandCache-Control: no-cacheso intermediaries do not cache or reshape the stream. - Forward, do not re-serialize. Pass provider chunks through as-is where you can; re-parsing and re-emitting every token adds latency and invites boundary bugs.
- Propagate cancellation. When the client disconnects, abort the upstream request too, or you keep paying for tokens no one will read. This is the server-side mirror of
AbortController.
A gateway is the natural home for this because it already terminates client connections and holds credentials. Registering each provider as a channel behind it lets the gateway stream from whichever upstream is healthy while presenting one consistent SSE shape to every client, and it logs a streamed request's token usage exactly like a non-streamed one. If you run many providers behind one endpoint, the streaming translation between their event formats is part of what an OpenAI-compatible API gateway does for you.
FAQ
Why does my JSON parser throw on the last line of the stream?
Because the final line of a Chat Completions stream is data: [DONE], and [DONE] is a literal sentinel, not JSON. Check for the [DONE] string and stop before you hand the payload to JSON.parse. This is the single most common streaming parse error.
Can I use the browser's EventSource for OpenAI-style streaming?
No. EventSource is GET-only and cannot set an Authorization header, while LLM calls are authenticated POST requests. Use fetch with res.body.getReader() and parse the text/event-stream yourself, which the example above shows.
How do I stop a stream and stop paying for tokens?
Attach an AbortController signal to the fetch and call controller.abort(). The connection tears down and the fetch promise rejects with an AbortError. Aborting only stops billing if the cancellation propagates all the way to the provider — in a proxied setup the server must abort its own upstream call too — so wire it to any UI "stop" control and to client disconnects, and propagate it end to end.
Do Chat Completions and Responses streams look the same on the wire?
They share the SSE transport but not the events. Chat Completions sends chat.completion.chunk deltas ending in [DONE]; the Responses API sends named semantic events like response.output_text.delta and response.completed. A parser written for one needs adjusting for the other.
Wire up a cancellable stream
Take the fetch + reader snippet above, point it at your gateway with stream: true, and render choices[0].delta.content to a text node as tokens arrive. Then add an AbortController and a stop button. Aborting is best-effort: your locally recorded output tokens should drop on a cancelled request, but treat that as a signal, not proof — the upstream may keep generating or billing briefly after you disconnect, and local counts can be estimated when the final upstream usage never arrives. To confirm the cancellation truly reached the provider, cross-check the gateway's stream_status (a client_gone marker) against the provider's own usage or invoice, not the local token count alone.