Streaming LLM Output: Tokens and Tool Calls

You send a question and watch a blank rectangle. Nothing. A little spinner, maybe. Four seconds, six, eight. Then the whole answer slaps onto the screen at once, three paragraphs you now have to start reading from the top. That’s the buffered experience, and it’s the default if you don’t do anything special.

Now the same model, same question, same eight seconds of total work. Except this time a word appears almost immediately, then another, then a sentence, and you’re already reading while the model is still writing the end. By the time it finishes you’ve half-consumed the answer. Nothing about the model got faster. You just stopped hiding the tokens.

Buffered

waiting for full response...

Streaming shows tokens as the model produces them instead of holding the whole answer back.

Streamed

Streaming shows tokens as the model produces them, so you read as it writes

Left: dead air, then a wall of text. Right: same total time, text you can start reading at once. The model generates token by token either way. Streaming is just the decision to show its work.

The whole input side of generative AI, retrieval, context assembly, the RAG plumbing, gets most of the attention. But the output side is where the user actually lives. And the mechanics are simpler than they look, right up until you try to stream a tool call, which is where it gets genuinely nasty. That last part is the point of this post.

Why bother: perceived latency is the real latency

An LLM is autoregressive. It produces one token, feeds it back in, produces the next, and repeats until it decides to stop. That loop runs whether or not you stream. The only question is when the tokens reach the user: after the last one is generated (buffered), or as each one comes out (streamed).

The metric that matters here is time to first token, TTFT. It’s the LLM’s version of time-to-first-byte: how long from “you hit send” to “the first readable thing appears.” Buffered responses have no meaningful TTFT, the first thing you see is also the last. Streaming splits one long wait into a short wait plus visible, readable progress. TTFT has become a standard enough concern that OpenTelemetry’s GenAI semantic conventions define gen_ai.server.time_to_first_token as a first-class metric, which is a good sign it’s a real SLO and not a vibe.

Bufferedtotal: 4.0s

nothing... nothing... (answer appears at the very end)

Streamedtotal: 4.0s

TTFT ~0.35s

first token, then a steady flow you're already reading

Same 4.0s of total generation. The buffered bar delivers everything at t=4.0. The streamed bar delivers something at t=0.35 and keeps going. Perceived latency is what people mean when they say an app "feels fast," and streaming wins it for free.

If you’ve read What Tokens Actually Are, the unit flowing across the wire here is exactly that token, one chunk of the model’s output at a time.

How it works over the wire: Server-Sent Events

The transport under almost every streaming LLM API is Server-Sent Events, SSE. It’s an old, boring, wonderfully reliable web standard, and boring is what you want under something that has to run for thirty seconds without hiccupping.

The shape: the client opens one long-lived HTTP connection, the server responds with Content-Type: text/event-stream, and instead of closing after one payload, it holds the connection open and pushes a sequence of events down it. Each event is plain text, optionally a named event: line and one or more data: lines, separated by a blank line. The client reads them as they arrive. That’s the entire protocol, and it’s specified in the WHATWG HTML standard, not some vendor’s blog post.

The Anthropic Messages API is a clean example of the event sequence. You get a message_start, then for each piece of the reply a content_block_start, a run of content_block_delta events carrying the actual tokens, and a content_block_stop. Then a message_delta with top-level metadata like stop reason, and finally message_stop. Text arrives as text_delta inside those content-block deltas. The repeated delta events in the middle are the stream, everything else is bookkeeping around them.

one open connection · text/event-stream● open

message_startrole, model, usage

content_block_startindex 0, type: text

content_block_deltatext_delta: "Stream"

content_block_deltatext_delta: "ing is"

content_block_deltatext_delta: " simple"

content_block_stopindex 0

message_stopdone

Events arriving one at a time over a single connection. The green content_block_delta rows are the tokens, everything around them is framing. Same connection stays open start to stop, no polling, no reconnect per token.

Every major cloud speaks a variant of this. AWS Bedrock’s ConverseStream gives you a unified, cross-model event set (messageStart, contentBlockDelta, messageStop), so the same parsing code works whether the underlying model is Claude, Llama, or something else. Google Vertex uses streamGenerateContent. Azure OpenAI is the familiar stream: true flag returning SSE chunks. Different names, same idea: one connection, a header of metadata, a middle of deltas, a footer that closes it out.

Aspect	Anthropic Claude	AWS Bedrock	Google Vertex	Azure OpenAI
Streaming call	Messages, stream:true	ConverseStream	streamGenerateContent	chat completions, stream:true
Transport	SSE (text/event-stream)	event stream (AWS)	SSE / chunked	SSE (text/event-stream)
Delta event	content_block_delta	contentBlockDelta	candidates[].content parts	choices[].delta

Four providers, one pattern. If you build your client around "open a stream, read framed deltas until stop," porting between them is mostly a rename job.

The hard part: streaming through an agent loop

Here’s where naive streaming falls apart. Streaming text is easy, each delta is a printable fragment, you append it and move on. But a model doesn’t only emit text. In an agent it emits tool calls, and a tool call has a structured JSON argument. The model streams that JSON the same way it streams prose: token by token, as fragments.

So the input to your get_weather tool doesn’t arrive as {"city": "Tokyo", "units": "c"}. It arrives as {"ci, then ty": "To, then kyo", "un, then its": "c"}. Those are input_json_delta events (partial JSON) under one content block. You cannot parse fragment three. It isn’t valid JSON. It isn’t even valid until the last fragment lands and the block stops.

The rule, and Anthropic’s fine-grained tool streaming docs are explicit about this: accumulate the partial-JSON fragments per content-block index, and only parse when that block’s stop event arrives. The index matters because a single turn can stream several tool calls interleaved or in sequence, and you must not mix fragment 2 of block 0 into block 1. Buffer by index, concatenate the pieces, and JSON.parse exactly once, at content_block_stop. Try to act on a half-formed call and you’ll either crash or fire a tool with garbage arguments.

input_json_delta (block index 0)

{"ci

ty": "To

kyo", "un

its": "c"}

Accumulated buffer

{"city": "Tokyo", "units": "c"}

content_block_stop → JSON.parse → call get_weather

Tool arguments stream as partial JSON. No single fragment is parseable. You concatenate by content-block index and parse exactly once, when the block stops. This is why streaming and tool use don't naively compose: text you can print immediately, a tool call you must wait to complete.

There’s a subtle trap even after you get the buffering right: you can accidentally add latency by buffering too much. LiveKit’s agent framework hit exactly this. Their LLM stream buffered tool-call deltas and, on every turn that involved a tool, introduced roughly 0.6 to 1 second of dead air before the agent spoke (issue #5826). The fix is to keep streaming the text portion to the user while you quietly accumulate the tool-call JSON in the background, rather than stalling the whole output until the JSON is complete. The lesson: buffering is necessary for the tool arguments, and poison for everything else. Stream what you can, buffer only what you must.

Structured output, the JSON-mode and schema-constrained responses you’d use in a RAG or extraction pipeline, streams the same way and needs the same buffer-then-parse discipline. If you’re wiring streaming into a real agent, the loop design in Designing an Agent That Doesn’t Go Off the Rails is where the tool-call handling actually lives.

The gotchas that show up in production

None of these are exotic. All of them will find you eventually.

Moderation on partial deltas

Content filters run mid-stream, so a chunk can get flagged after you've already shown the user the tokens before it. You need a plan for retracting or halting a stream that's turned unsafe partway through, not just filtering the final blob. Azure's content-streaming docs cover the asynchronous-filter tradeoff directly.

Proxies that buffer your stream to death

Load balancers, reverse proxies, and CDNs love to buffer responses for efficiency, which quietly re-assembles your SSE stream into one big delayed blob. All your TTFT work, gone. You have to explicitly disable response buffering on the path (think nginx proxy_buffering off), or the stream never reaches the client as a stream.

Reconnection and Last-Event-ID

SSE has built-in reconnect: the browser remembers the last event id and sends Last-Event-ID on reconnect so the server can resume. That only works if your server actually emits ids and knows how to pick up mid-generation. Most LLM streams don't resume gracefully, so decide up front whether a dropped connection means retry-from-scratch or resume.

The ~6-connection-per-domain limit

Over HTTP/1.x, browsers cap concurrent connections per domain at around six, and a held-open SSE stream eats one for its whole lifetime. A few open streams plus normal traffic and you're blocked. HTTP/2 multiplexes and mostly dissolves this, which is one more reason to serve streams over HTTP/2.

Streaming is boring to get working and finicky to get right. Every one of these is a "worked in dev, mysteriously broke in prod behind the load balancer" story waiting to happen.

The takeaway

Streaming doesn’t make your model faster. It makes your model honest about the fact that it’s already writing one token at a time, and it hands those tokens to the user the instant they exist instead of hoarding them for a dramatic reveal nobody asked for. The transport is a thirty-year-old web standard doing exactly what it was built to do. The perceived-latency win is basically free.

The one place it stops being free is the agent loop, where tool-call arguments arrive as partial JSON that you have to buffer by index and parse only when the block completes, streaming the prose while quietly accumulating the structure. Get that split right and you get an agent that talks while it thinks. Get it wrong and you either crash on half-formed JSON or add a second of dead silence to every tool call, which, as LiveKit found out, users notice immediately. Show the tokens. Buffer only the JSON. Ship it over HTTP/2 and turn off the proxy buffering before you wonder why none of it works.

This is one of a set of streaming posts, the streaming backbone under generative AI and streaming context into LLMs and agents, and it sits alongside the broader three AI systems, same design series.