Designing a RAG System That Actually Retrieves
Someone in your company types “what’s our refund window on annual plans?” into a chat box and hits enter. They want one sentence back, correct, with a link to the actual policy doc, and they want it before they’ve finished reaching for their coffee. The policy changed last Tuesday. There are four thousand documents in the system and this person can only legally see about six hundred of them.
That’s the job. Not “put an LLM on top of a vector store.” Anyone can do that in an afternoon and demo it and feel great. The demo answers the three questions you tested it on and then someone asks the fourth question and it confidently cites a policy that was deleted in March.
This is post two in the AI System Design on the Cloud series. In the first post I made the case that RAG, agents, and recommendations are the same funnel: cheap wide recall, then expensive narrow precision, then policy. This post works the RAG funnel out in full, box by box, and names the cloud service that fills each box. The whole time, keep one thing in mind: the model is rarely the problem. Retrieval is.
Why RAG exists at all
Quick sanity check, because people reach for RAG reflexively and sometimes they shouldn’t.
You use retrieval-augmented generation when three things are true at once. Your corpus is too big to stuff into a context window, thousands of docs, no contest. Your content changes too often to bake into the model, you can’t retrain because someone edited a pricing table this morning. And you need citations, an answer with no source is useless when the question is “what’s our policy,” because nobody will trust it and legal will have opinions.
RAG solves all three by keeping the knowledge outside the model. The docs live in an index. At question time you go find the relevant pieces and hand them to the model as context. Edit a doc, re-index that doc, done. The model never memorized anything, so it can’t be out of date, as long as your index isn’t.
That “as long as your index isn’t” is doing a lot of work. Hold that thought.
The two pipelines
The mistake beginners make is thinking of RAG as one flow. It’s two, and they run at completely different times.
The offline path is where quality is quietly won or lost. It runs on a schedule or on a change event, and it’s allowed to be slow. The online path runs on every keystroke-ish and has a latency budget measured in a second or two. If you only optimize the online path, you’re polishing the wrong pipeline. Let’s walk the offline one first, because that’s where the retrieval you’re so worried about actually gets built.
Chunking: where good RAG is won or lost
You can’t embed a fifty-page document as one vector. It’d be mush, an average of everything, specific about nothing. So you split it into chunks. The naive way is to count characters: every 500 tokens, cut. Simple, and it quietly wrecks your retrieval.
Fixed-size split (every 500 tokens)
The cut lands wherever the counter hits 500. A sentence gets sliced in half. A table's header ends up in one chunk and its rows in the next, so neither chunk means anything on its own.
Structure-aware split
Cuts follow the document's own seams: headings, paragraphs, table boundaries. Each chunk is a complete thought. A little overlap between chunks means a sentence spanning a boundary still survives in one of them.
Two more moves make chunking genuinely good. Overlap: let consecutive chunks share a little text, so an idea that straddles a boundary lives intact in at least one of them. And contextual retrieval: before you embed a chunk, prepend a one-line summary of the section and document it came from. “From the Refunds section of the 2026 Annual Plan Terms:” glued to the front of a chunk about the 30-day window turns an ambiguous fragment into something that retrieves cleanly, because now the chunk carries its own context instead of assuming the reader remembers what section they’re in. It costs a bit at index time and pays you back on every query.
If you get one thing right in the whole system, get chunking right. A great model reranking bad chunks is a great chef working with spoiled ingredients.
Retrieval: dense and sparse, together
Now the part everyone thinks is RAG. You’ve got the question, you need the right chunks. There are two ways to search, and the trap is picking one.
Dense (semantic) search embeds the query and finds chunks whose vectors are nearby. It understands meaning. “How long do I have to get my money back” finds the refund-window chunk even though it shares almost no words with it. But dense search is fuzzy about exact tokens. Ask about error code TLS_0x8F or SKU AX-2200 and the embedding blurs it into “some error code, some product,” and you get near-misses.
Sparse (keyword / BM25) search matches literal terms. It nails the error code, the SKU, the acronym, the exact product name. But ask it a paraphrase and it shrugs, because none of the words line up.
So you run both and fuse the rankings, usually with reciprocal rank fusion, which is a fancy name for a simple, robust way of blending two ranked lists without having to calibrate their scores against each other. Hybrid retrieval isn’t a nice-to-have. In a real corpus full of product names and codes and jargon, it’s the difference between “usually finds it” and “finds it.”
And this is exactly the stage where permissions belong. More on that in a second.
Rerank: cheap and wide, then expensive and narrow
Hybrid search gives you a decent ranked list, but “decent” isn’t good enough to feed the model. The top result by vector similarity is often the third-best answer to the actual question. So you add a second, smarter pass.
The bi-encoder in the retrieval step encodes the query and every chunk separately and compares vectors, which is fast because the chunk vectors were computed offline. A cross-encoder reranker is different: it takes the query and one chunk together, as a pair, and reads them jointly. That joint reading is dramatically more accurate at judging “does this chunk actually answer this question,” and it’s also far too slow to run against ten thousand chunks per query. So you don’t. You let cheap retrieval narrow ten thousand down to fifty, then spend the expensive cross-encoder only on those fifty, and hand the model the top eight.
That’s the funnel from post one, sitting right in the middle of RAG. Cheap wide net, expensive narrow judgment, and it’s forced on you by the same constraint every time: your best step is too slow to run on everything.
Permissions belong in retrieval, not generation
Here’s the mistake that turns a clever system into a data-leak incident. It’s tempting to retrieve everything, generate an answer, and then check whether the user was allowed to see the sources. That’s backwards. By the time you’re filtering the output, the model has already read documents this person can’t access, and even a well-behaved model can leak the gist of a restricted doc in a paraphrased answer.
Permissions have to be a metadata filter at retrieval time. Every chunk carries the access tags of its source document. When the query runs, you filter the candidate set to only what this user can see before anything gets ranked or read. The model never touches a document the person isn’t cleared for, so there’s nothing to leak. Filter at the door, not on the way out.
Grounding, citations, and the courage to say “I don’t know”
Now the model finally speaks. Its instructions are narrow on purpose: answer only from the chunks provided, cite which chunk each claim came from, and if the chunks don’t contain the answer, say so instead of guessing.
That last part is the hard cultural sell and the most important. A RAG system that says “I don’t have that in the docs” when it genuinely doesn’t is worth ten times more than one that produces a fluent, confident, wrong answer. Confident and wrong is the failure mode that erodes trust and, in a policy context, gets people into real trouble. Prefer the honest miss. Make “I don’t know” a first-class, encouraged output, not an embarrassment the prompt tries to avoid.
Citations do double duty here: they let the user verify, and they let you audit. When an answer is wrong, the citation tells you instantly whether retrieval handed the model bad chunks or the model ignored good ones. Which brings us to the thing most teams skip.
Caching, because you’ll answer the same thing a thousand times
People ask the same questions. “What’s the refund policy,” “how do I reset my password,” phrased forty different ways. Three layers of cache save you real money and latency:
Exact-match cache for identical queries, trivial and instant. Semantic cache that recognizes “how do I get a refund” and “what’s your refund process” as the same question via embedding similarity, and returns the stored answer without re-running the whole pipeline. And prompt / context caching at the model layer, where the provider caches the fixed part of your prompt (system instructions, retrieved context that repeats) so you’re not paying to re-process the same tokens every call. Stack all three and a big chunk of your traffic never hits the expensive path.
The same design, filled in three ways
None of this requires building from scratch. Every cloud has a managed service for each box. Here’s the map as of mid-2026.
| Pipeline box | AWS | GCP | Azure |
|---|---|---|---|
| Parse / OCR | Textract | Document AI | Document Intelligence |
| Managed RAG | Bedrock Knowledge Bases | Vertex AI Search | Azure AI Search (integrated vectorization) |
| Embeddings | Titan or Cohere on Bedrock | gemini-embedding | Azure OpenAI embeddings |
| Vector store | OpenSearch Serverless, Aurora pgvector, or S3 Vectors | Vertex Vector Search or AlloyDB pgvector | Azure AI Search or Cosmos DB |
| Reranker | Bedrock Rerank (Cohere) | Vertex Ranking API | AI Search semantic ranker |
| LLM | Bedrock (Claude) | Vertex (Gemini) | Microsoft Foundry (GPT / Claude) |
| Guardrails | Bedrock Guardrails | Model Armor | Azure AI Content Safety |
A word of caution on the “Managed RAG” row. Bedrock Knowledge Bases, Vertex AI Search, and Azure AI Search will each do chunking, embedding, retrieval, and reranking behind one API, and that’s a genuinely good way to ship a first version fast. The catch is that the moment your retrieval quality plateaus, you’ll want to reach in and change the chunking strategy or swap the reranker, and how much the managed service lets you do that varies a lot. Start managed, but know which box you’ll want to pry open first.
The gotchas nobody demos
The demo works. Then it meets reality. Three failures show up in basically every RAG deployment, and none of them are the model’s fault.
A doc gets edited or deleted, but its old chunks linger in the index. The bot confidently cites a policy that no longer exists. Retrieval did its job perfectly, it found the chunk, the chunk was just a ghost. Fix: delete and upsert on every change, and stamp chunks with version metadata so superseded content can be filtered out.
The right chunk never made it into the top-K. So the model, handed nothing useful, falls back on what it half-remembers from training and produces a fluent, authoritative, wrong answer. It looks like a hallucination. It's a retrieval failure wearing a hallucination costume. Fix: measure retrieval recall separately, so you catch the miss before the user does.
You upgrade the embedding model for better quality. But your index was built with the old one, and vectors from two different models don't live in the same space, so queries and chunks stop lining up and retrieval quietly degrades. Fix: changing the embedding model means re-embedding the entire corpus. Pin the model, and treat a change as a full reindex, never a hot swap.
Evaluate retrieval separately from generation
That second gotcha points at the single most useful discipline in RAG, so it gets its own section: measure retrieval and generation as two different things.
Teams love to evaluate the final answer, thumbs up, thumbs down, “was this helpful.” That tells you something went wrong but not where. Split it. Measure retrieval on its own with recall@k (did the right chunk make it into the top k?) and MRR (how high up was it?). Measure generation on its own with groundedness / faithfulness (did the answer stick to the retrieved chunks, or did it invent?).
The number that matters most: most RAG failures are retrieval failures. If recall@k is low, no model on earth can save the answer, because the answer was never in the room. Fix retrieval first, always.
When you split the metrics, debugging stops being guesswork. Low recall means work on chunking, hybrid search, or reranking. High recall but low groundedness means the model is ignoring good context, so tighten the prompt or the model. Without the split, you’ll spend a week tuning prompts to fix a problem that lived in your chunker.
The takeaway
RAG is not “an LLM with a vector database.” It’s a retrieval system that happens to end in a language model, and almost all of the engineering, and almost all of the failure, lives in the retrieval half. Get the chunks right, search dense and sparse together, funnel wide-then-narrow through a reranker, filter permissions at the door, ground every claim in a citation, and measure the retrieval step on its own so you know where it breaks.
Do that and the model’s job becomes almost boring: read eight good chunks, write one honest sentence, cite the source. Boring is exactly what you want. The magic was never in the model. It was in making sure the right paragraph showed up before the model ever opened its mouth.
Next in the series: designing the agent, where the same funnel drives a loop instead of a single pass, and the recommendation system, the funnel in its purest, highest-scale form. And if you skipped it, the first post is the map for why all three share this shape.