The Streaming Backbone Under Generative AI
A load test hits a demo I once watched go sideways. Not the model, the model was fine. The moment traffic tripled, the app started calling the embedder in its own request path, one embed per incoming event, synchronously. The embedder throttled, the requests backed up, the whole thing fell over in about forty seconds. The fix wasn’t a bigger model or a faster GPU. It was a pipe. A stream in the middle that soaked up the burst and let the embedder work at its own pace.
That pipe is the part nobody draws. Every generative-AI architecture diagram puts the LLM in the center with arrows pointing at it, and treats those arrows as if they’re free. They aren’t. They’re the streaming backbone, and it’s doing more work than the model on a bad day.
What the stream actually does for GenAI
Strip away the vendor names and a stream is one thing: a durable, ordered, replayable log sitting between the things that produce events and the things that consume them. Producers append. Consumers read at their own pace and remember where they were. Nobody blocks anybody. In a generative-AI system it quietly does three jobs.
- Feeds live context into LLMs. Real-time grounding. A prompt built from what’s happening now, the last few minutes of alerts, the current order state, this session’s clicks, instead of whatever was true at deploy time.
- Keeps RAG indexes fresh. Every change to a source becomes an event, gets re-embedded on the way through, and upserts a single vector so retrieval reflects reality. (That’s its own post, the companion piece on streaming into RAG.)
- Moves events between agents. Agent-to-agent messaging. One agent emits a result, others react, without every agent holding a direct HTTP connection to every other agent.
Same pipe, three jobs. Here’s its shape.
The backbone, end to end
The log is the interesting box. It’s not a queue that deletes on read, it’s an append-only log split into partitions, and each record has a position (an offset in Kafka, a sequence number in Kinesis). Records aren’t removed when a consumer reads them, they age out on a retention policy. So a slow consumer can catch up later, a new consumer can replay from the beginning, and a crashed one resumes from its last committed offset instead of losing everything. That single property, replayability, is what makes the backbone safe to build a GenAI pipeline on.
Four clouds, one idea, slightly different vocabulary. Kinesis gives you shards, partition keys, and sequence numbers. Kafka gives you topics, partitions, offsets, and consumer groups. Pub/Sub gives you topics with ordering keys. Event Hubs gives you partitions, consumer groups, and checkpoints, and it speaks the Kafka protocol so a lot of Kafka tooling just works against it. Learn one, and the others are mostly renaming.
Ordering is only a per-partition promise
Here’s the thing that bites people. A stream does not guarantee global order. It guarantees order within a single partition, and that’s it. Which means where a record lands is a decision you make, through the partition key.
Same key, same partition
All of order-42's events route to one partition. They stay in order. The consumer sees created, then paid, then shipped, never shuffled.
Different keys, parallel
Different orders spread across partitions and process in parallel. Fast, and fine, because you never needed order-42 and order-77 ordered against each other.
Why a GenAI system cares: out-of-order events quietly corrupt state. If two edits to the same document arrive out of order, the older embedding can overwrite the newer one, and now your fresh vector is silently stale. If an agent watching an order sees “shipped” before “paid,” it reasons off a state that never existed. The rule is boring and load-bearing: route all events for one entity to the same partition by using the entity’s ID as the partition key. Everything for that entity stays ordered, everything across entities still runs in parallel. You get correctness where you need it and throughput everywhere else.
Delivery semantics: pick your poison
When you wire a producer to a stream to a consumer, exactly-once sounds like the obvious thing to want. It’s rarely what you want to pay for. There are three modes, and they trade the same two failures against cost.
At-most-once
Fire and forget. On failure, the record is just gone. Fastest, cheapest, and it loses data. Fine for lossy telemetry, wrong for anything that matters.
risk: silent lossAt-least-once
Retry until acknowledged. Nothing is lost, but a retry after a lost ack means the same record shows up twice. The common default, and duplicates are manageable.
risk: duplicatesExactly-once
Idempotent producer plus transactions dedupe end to end. No loss, no dupes, at a throughput cost (Confluent measures roughly 3% on Kafka). Real, not free.
cost: ~3% throughputFor generative AI there’s a pragmatic combo that beats reaching for exactly-once by reflex: at-least-once delivery plus an idempotent upsert-by-ID on the consumer. Think about what a duplicate actually does in a GenAI pipeline. A repeated “document 42 changed” event just re-embeds the same text and upserts it to the same vector ID. The second write lands on top of the first and produces the identical result. Harmless. You made the operation idempotent at the sink, so you don’t need the stream to guarantee no duplicates. You get exactly-once effects without paying for exactly-once delivery. (Worth noting the 3% figure comes from Confluent, who are Kafka-aligned, but the shape of the tradeoff holds regardless of vendor.)
Back-pressure is the reason the pipe exists
Remember the demo that fell over in forty seconds. This is the visual of why.
Back-pressure is the polite word for “the consumer is slower than the producer, so slow the producer down.” Without a stream, there’s nowhere to put the overflow, so the pressure lands on whatever is downstream, usually your embedder or your model endpoint, and it throttles you or runs up a bill you didn’t budget for. With a stream, the log is a buffer. Producers keep appending, the log holds the backlog, and consumers pull at whatever rate they can sustain. A traffic spike becomes a temporary rise in lag instead of an outage. This is the single biggest reason a GenAI system in production has a stream in the middle rather than a direct call. Models and embedders have rate limits and real per-token cost. You do not want a viral moment translating directly into throttled requests or a five-figure invoice.
Windowing: aggregating before you spend a token
Sometimes you don’t want to feed the LLM every event. You want a summary of a slice of time. “Summarize the last five minutes of alerts.” That’s a window, and the stream-processing layer (Apache Flink and friends) gives you two shapes of it.
A tumbling window chops time into fixed, non-overlapping buckets: every five minutes, take everything that arrived and produce one summary. Clean, one output per bucket, and each event counts once. A sliding window is a fixed span that advances by a smaller step, so windows overlap and an event can belong to several. You’d use tumbling for “give me a fresh five-minute digest every five minutes” and sliding for “keep me a rolling five-minute view that updates every minute.” Either way, the point for GenAI is the same: you aggregate the raw firehose down to something worth a prompt before you spend tokens on it, instead of calling the model once per raw event.
The stream-processing layer
The log holds events. Something has to transform them before the GenAI step, chunk text, compute an embedding, aggregate a window, filter noise. That’s the stream-processing layer, and it sits directly on the stream: Apache Flink, its managed forms (AWS runs a Managed Service for Apache Flink), or Kafka Streams if you’re in Kafka’s world. AWS publishes the canonical shape of this for GenAI: Kinesis to Managed Flink to Bedrock (Claude) to OpenSearch. Events land in Kinesis, Flink reads and windows and enriches them, calls Bedrock to run a model or an embedding, and writes the result to OpenSearch for retrieval. Every box is doing one job, and the stream is what lets them run at independent speeds without one starving or drowning the next.
Events between agents, not HTTP between agents
The newest job for the backbone is the most interesting. As soon as you have more than a couple of agents talking to each other, the naive design (every agent holds a direct HTTP connection to every other agent) turns into a mess. Ten agents wired point to point is up to ninety directional connections to build, secure, and debug, and it grows with the square of the count. Add one agent and you touch every other.
Put a stream in the middle and it goes linear. Each agent publishes its results to a topic and subscribes to the topics it cares about. Nobody needs a direct line to anybody. An agent can be added, restarted, or scaled without rewiring the others, and because it’s a durable log, a new agent can even replay history to catch up on what it missed. This is the emerging pattern behind event-driven multi-agent systems, and it’s why people connecting agent-to-agent protocols like A2A and MCP keep landing on Kafka as the broker underneath: the coordination problem between agents is the same pub/sub problem the streaming world solved a decade ago. (Confluent has written the loudest version of this argument, and yes, they sell Kafka, but the quadratic-to-linear point stands on its own.)
The same backbone across clouds
Four vendors, one mental model. Here’s the concept map so a diagram on one cloud reads on any other.
| Concept | Kafka / Kinesis | Pub/Sub | Event Hubs |
|---|---|---|---|
| The stream | Topic / stream | Topic | Event hub (topic) |
| Ordering unit | Partition / shard (+ key) | Ordering key | Partition (+ key) |
| Delivery guarantee | At-least-once, exactly-once opt-in | At-least-once, ordered opt-in | At-least-once, checkpointed |
| Managed offering | MSK / Confluent / Kinesis | Pub/Sub (fully managed) | Event Hubs (fully managed) |
| Stream processing | Flink / Kafka Streams | Dataflow (Beam) | Stream Analytics / Flink |
The takeaway
The model gets the credit. The backbone keeps it standing. Every serious generative-AI system, real-time grounding, live RAG, multi-agent coordination, has a durable ordered log in the middle doing the unglamorous work: absorbing spikes so a viral moment doesn’t throttle your endpoint, preserving per-entity order so a stale vector never overwrites a fresh one, and letting agents talk through topics instead of a spaghetti of HTTP. Partition by entity key for the ordering you need. Lean on at-least-once plus idempotent upserts and skip paying for exactly-once you don’t. Let the log take the back-pressure. Do that and the interesting failure mode stops being “the whole thing fell over in forty seconds” and starts being “lag went up for a minute, then recovered.” Which is the entire point.
This is one of three siblings on the streaming backbone. This post is the plumbing; Streaming Context Into LLMs and Agents covers feeding live context in on the read side, and Streaming LLM Output: Tokens and Tool Calls covers streaming the model’s output back out. Read together they’re the full round trip.
References
I wrote the argument and drew every diagram from scratch. The patterns aren’t mine to claim, though, and the docs and posts below are where the concepts are specified and where the AWS reference pipeline lives. Nothing here is copied from them.
Part of the AI System Design on the Cloud series. For the systems this backbone feeds, see Designing a RAG System That Actually Retrieves and Designing an Agent That Doesn’t Go Off the Rails. The direct companion is Streaming Data Into RAG: Keeping the Index Live.