Streaming Data Into RAG: Keeping the Index Live
If you’ve read the RAG series here, you know the machinery: chunk your documents, embed them, drop the vectors in a store, and at query time retrieve the closest ones and hand them to the model. It works. But there’s a quiet assumption baked into every diagram of it, and almost nobody says it out loud.
The index is a photograph.
You embed your documents once. From that second, the vectors are frozen. The world keeps moving, tickets get updated, prices change, an order ships, someone edits the policy, and your index sits there holding a picture of how things were at the last reindex. Ask the bot “what’s the status of order 4471?” and it cheerfully answers from last night’s snapshot. Confidently. Wrongly.
For a policy handbook that changes twice a year, a nightly batch reindex is completely fine. For anything that actually changes, it’s a slow-motion correctness bug. This post is about closing that gap: wiring a stream of changes straight into the vector index, so retrieval reflects now.
Snapshot RAG vs live RAG
Snapshot RAG
Embed the whole corpus in a batch job. Re-run it on a schedule, nightly, hourly, whenever. Between runs, the index is a fixed picture. Simple, cheap, and correct only until the moment the first document changes.
freshness = time since last batch
Live RAG
Every change to a source becomes an event. The event flows through a stream, gets re-chunked and re-embedded, and updates just that item in the index within seconds. The picture is never more than a moment old.
freshness = seconds
Notice what does not change: the query path. Retrieval, reranking, the model, all of it stays exactly as the design post laid it out. Streaming only rewrites the ingestion side, the offline path that keeps the index current. That’s the whole trick. You’re not building a different RAG system. You’re replacing the batch loader with a live one.
The use cases that actually need this
Before the pipeline, the honest question: do you need it? Streaming ingestion is more moving parts and more cost, so reach for it only when staleness is a real bug. It usually is when the corpus is one of these:
- Support and operations. A bot answering over live tickets, order status, account state. “It shipped an hour ago” needs to be retrievable an hour ago, not tomorrow.
- Commerce. Prices, inventory, availability. Retrieving “in stock” for something that sold out twenty minutes back is a customer-trust problem.
- Conversations and collaboration. Chat logs, docs, comment threads that people expect the assistant to have already read.
- Signals and telemetry. Logs, metrics, fraud events, IoT readings, anything where the useful window is minutes wide.
If your corpus is contracts, manuals, or published articles, close the tab and keep your nightly batch. You don’t need any of what follows.
The pipeline, AWS in the loop
Here’s the shape end to end. A change happens at the source, and instead of waiting for a batch job to notice, you turn it into an event and push it through a stream that ends at the vector store.
Walking the boxes, and why each one is what it is:
The change, captured. The cleanest source of “something changed” is change data capture. If your data lives in DynamoDB, DynamoDB Streams hands you an event for every write, no polling. A relational database gives you CDC through Database Migration Service or Debezium onto a stream. A document dropped in S3 fires an EventBridge or S3 notification. The point is the same: don’t scan the whole corpus looking for diffs, let the source tell you exactly what moved.
The stream itself. This is the load-bearing part, and it’s why streaming beats “just call the embedder in your write path.” Kinesis Data Streams (or MSK, managed Kafka, if you want Kafka semantics) sits between the firehose of changes and your embedding step. It buffers, so a burst of updates doesn’t melt your embedder. It’s durable, so if the consumer dies mid-batch, the events are still there. And it preserves order within a shard, which matters more than you’d think, we’ll get to that. Without a stream, a spike in source changes goes straight at Bedrock and either throttles or bankrupts you.
Chunk and embed. A Lambda function consumes from the stream in small batches. For each changed item it does the same work your batch job did, but for one thing: re-chunk it (structure-aware, with overlap, exactly as Part 2 of the RAG series argued), then call Bedrock to embed each chunk. Lambda is the right tool here because the work is bursty and embarrassingly parallel: scale to zero when nothing’s changing, fan out when the stream floods. Watch cold starts on the latency-sensitive edge, but for ingestion a few hundred milliseconds is nothing.
Upsert, not append. The last box is where most people quietly introduce a bug, so it gets its own section.
The upsert trap
The instinct, when a new event arrives, is to embed it and add it to the index. Add. That’s the bug.
Append (wrong)
Upsert by stable ID (right)
If you append, your index becomes a graveyard of every version a document ever had, and the stale ones outnumber the fresh. The bot retrieves the old price and the new one and picks whichever the reranker liked. The fix is boring and essential: give every chunk a stable, deterministic ID derived from the source’s primary key plus the chunk index, and upsert on it, replace if it exists, insert if it doesn’t. And handle deletes as first-class events: when the source row is deleted, the stream carries a delete, and your consumer removes those chunk IDs. An index that can’t delete is an index that lies eventually.
Why the freshness gap is the whole point
With a nightly batch, the honest description of your index is “correct as of 2am, degrading until tomorrow.” At 5pm it’s fifteen hours stale in the worst case. You can shrink that by reindexing hourly, but hourly reindex of a big corpus is expensive and wasteful, you re-embed millions of unchanged chunks to catch the handful that moved. Streaming flips the economics: you only ever embed what actually changed, and the freshness is measured in seconds, not hours. You do more small work and far less total work.
The gotchas nobody warns you about
Streaming ingestion has a specific set of ways to hurt you. None are dealbreakers, all are worth designing for up front.
Out-of-order events. Two edits to the same item can arrive out of order, and now your "current" vector is actually the older one. Order within a Kinesis shard is guaranteed, so partition by the item's key so all its edits land on the same shard. Or carry a version or timestamp on each event and refuse to overwrite a newer vector with an older one.
Embedding cost per event. Every change triggers an embed call, and a chatty source (a row that updates on every click) can turn into a Bedrock bill you didn't plan for. Debounce: collapse rapid repeated edits to the same item, and only re-embed when the text that actually gets embedded changed, not on every metadata tweak.
Back-pressure and poison events. A traffic spike or a single malformed record can stall the consumer. Let the stream buffer the spike (that's its job), set sane batch sizes and concurrency on the Lambda, and send records that fail repeatedly to a dead-letter queue instead of blocking the shard behind them forever.
Embedding-model drift, now live. The old rule still holds: change the embedding model and old vectors are incompatible with new queries. In a streaming world you can't just "re-run the batch." You need a reindex strategy, usually a parallel index you backfill and cut over to, because the live one never stops taking writes.
That last one is the sharp edge of live systems generally: there’s no quiet window where nothing is happening. Every migration is a migration under load. Plan for a shadow index and a cutover, not a maintenance window.
The same pipeline on the other clouds
The pattern is cloud-agnostic, stream the changes, embed on the way through, upsert by ID. Here’s how the boxes map if you’re not on AWS:
| Box | AWS | GCP | Azure |
|---|---|---|---|
| Change capture | DynamoDB Streams / DMS CDC | Datastream / Firestore triggers | Cosmos DB change feed |
| The stream | Kinesis / MSK | Pub/Sub | Event Hubs |
| Chunk + embed | Lambda | Dataflow / Cloud Functions | Azure Functions |
| Embedding model | Bedrock (Titan / Cohere) | Vertex (gemini-embedding) | Azure OpenAI embeddings |
| Vector store | OpenSearch / S3 Vectors | Vertex Vector Search / AlloyDB | Azure AI Search / Cosmos DB |
The takeaway
A batch-loaded RAG index answers questions about the past and pretends it’s the present. For a lot of corpora that’s fine. For the ones that move, the fix isn’t a smarter model or a bigger context window, it’s admitting the index has to be fed, not photographed.
Streaming ingestion is that feeding tube. A change becomes an event, the event rides a durable stream, gets embedded once on the way through, and upserts a single item in the index within seconds. The query side never even knows. And the two things that turn it from a demo into a system you can trust are the least glamorous parts: upsert by stable ID, and handle deletes. Get those right and your bot stops answering yesterday’s questions with yesterday’s data.
References
I built the argument and the diagrams from scratch, but the pattern isn’t mine to claim: AWS publishes an official reference architecture for exactly this, and the surrounding docs are worth reading if you’re implementing it. Everything below is a link, nothing here is copied from them.
This extends the RAG thread: start with what RAG really is, then chunking, then why it still fails. For the full serving-side design and the cloud service map, see Designing a RAG System That Actually Retrieves, part of the AI System Design on the Cloud series.