The Three AI Systems That Are Secretly the Same Design
I’ve built a RAG chatbot, wired up a tool-calling agent, and worked on ranking. For a long time I filed them under three separate headings in my head. Different problems, different diagrams, different words.
Then one afternoon I was drawing the RAG pipeline on a whiteboard, retrieve a bunch of chunks, rerank them down, hand the good ones to the model, and I realised I’d drawn the recommendation system from the week before. Same funnel. Different labels.
Once you see it you can’t unsee it. RAG, agents, and recommendations are the same machine wearing three outfits. And the shape of that machine tells you exactly where each one is going to break.
The shape
Here’s the thing all three do. You have far too many candidates to look at carefully, millions of documents, millions of products, an open-ended space of possible next actions. You can’t run your expensive, smart step on all of them. So you don’t.
Instead you build a funnel.
Three stages, and the trick is that the cost per item goes up as the number of items goes down.
That’s it. That’s the whole design. Now watch it show up three times.
The same funnel, three times
| Stage | RAG chatbot | Agent | Recommendations |
|---|---|---|---|
| Cheap wide recall | Hybrid vector + keyword search over chunks | The planning step: what could I do next? | Two-tower retrieval, millions to hundreds |
| Expensive precision | Cross-encoder rerank, then the LLM answers | The reasoning step: read results, decide | Heavy ranking model scores each candidate |
| Policy layer | Groundedness check, PII filter, citations | Guardrails, human-in-the-loop on risky actions | Diversity, freshness, ads, dedupe |
Look at the middle column. A RAG chatbot casts a wide net with cheap vector search (you can search millions of chunks in milliseconds), narrows it with an expensive cross-encoder reranker that reads the query and each chunk together, then hands a tidy handful to the model and checks the answer is grounded before it ships.
An agent does the same thing in a loop. The planning step is cheap-ish recall over “things I could do next.” The actual reasoning, deciding which tool to call and reading what came back, is the expensive part, and you don’t want to do more of it than you have to. Guardrails and human approval gates are the policy layer sitting on top of every risky step.
A recommendation system is the textbook version. Candidate generation retrieves a few hundred items out of millions using cheap nearest-neighbour lookups. A ranking model then scores just those few hundred with rich features. Then a re-ranking pass handles diversity and business rules, so you don’t show someone ten near-identical things or bury the sponsored slot.
Three systems. One funnel. Cheap-wide, then expensive-narrow, then policy.
Why it keeps happening
This isn’t a coincidence, and it isn’t a fashion. It falls out of a hard constraint: your best step is too expensive to run on everything.
A cross-encoder that reads a query and a document together is far more accurate than comparing two embeddings, but it’s also orders of magnitude slower, so you can’t run it on ten million documents per query. A frontier model reasoning over tool results is powerful and pricey, so you want to call it as few times as possible. A deep ranking model with hundreds of features gives great scores but you can’t score a whole catalogue with it in the eighty milliseconds you’ve got.
So every one of these systems arrives at the same compromise. Use something cheap and roughly-right to throw away 99% of the candidates, then unleash the expensive thing on what’s left. The funnel isn’t a design choice someone made. It’s what you’re forced into the moment quality and scale pull in opposite directions.
The one-line version: when your smartest step is too slow to run at scale, you build a funnel in front of it. RAG, agents, and recommendations are three funnels feeding three different expensive steps.
Why this is useful, not just tidy
Noticing the shared shape would be a fun trivia fact if it stopped there. It doesn’t. The shape tells you where these systems fail, and it’s the same failure every time.
Most failures are recall failures, not precision failures. Stage 2 can only pick from what stage 1 handed it. If the right document never made it into the retrieved pool, the world’s best reranker and the smartest model will confidently answer from the wrong context. If the right product never got retrieved, no ranking model can surface it. If the agent never considered the right action, no amount of reasoning saves the plan.
So when a RAG bot gives a wrong answer, the instinct is to blame the model or tweak the prompt. Usually the model is fine. The retrieval missed. The single most useful thing you can do across all three systems is measure the recall of stage 1 separately from the quality of stage 2. How often does the good candidate actually make it into the pool? That number, not the eloquence of the final output, is where most of your quality lives.
The second shared lesson: stage 3 is not optional, and it’s not the model’s job. Diversity, safety, freshness, permissions, none of these are things your accuracy metric rewards, so your model will never learn them on its own. You bolt them on as an explicit policy layer, or they don’t happen. A ranking model left to itself will happily show ten versions of the same item. A RAG model left to itself will happily quote a document the user isn’t allowed to see. The policy layer exists precisely because being accurate and being right for the product are different things.
Start with the cheapest funnel that works
There’s a corollary worth saying out loud, because it saves a lot of pain. Not every problem needs the full three-stage machine.
If a single model call answers the question, that’s your whole system. If a fixed sequence of steps does the job, build the sequence, a RAG pipeline is a fixed chain, not an agent, and that’s a feature. You only reach for the elaborate version, the multi-source retrieval, the agent loop, the full candidate-generation-and-ranking stack, when the scale genuinely forces the funnel on you.
The mistake I see most often is people building stage 1 and stage 2 for a problem that had five hundred candidates, where you could just run the expensive step on all of them and skip the funnel entirely. The funnel is what you build when you can’t do that. Know which situation you’re in.
The upshot
RAG, agents, and recommendations aren’t three things to learn. They’re one thing, a funnel that trades cheap breadth for expensive depth, pointed at three different expensive steps. Learn the shape once and each new system becomes “oh, it’s the funnel again, feeding a different thing.”
And when one of them misbehaves, you already know where to look first. Not the clever model at the narrow end. The wide, cheap net at the top, quietly dropping the answer before anything smart ever got to see it.
The next posts in this series work each funnel out in full: the RAG design, the agent design, and the recommendation design, box by box, with the cloud services that fill each one on AWS, GCP, and Azure.