Designing a Recommendation System: The Funnel
You know the moment. You finish a video, and the “up next” is somehow exactly the thing you didn’t know you wanted. Or you open a shopping app and the home feed leads with the one product you’d been half-thinking about. It feels like the app read your mind.
It didn’t. What it did was pick a handful of items out of millions, and it did it in the time the page took to paint. Tens of milliseconds. That constraint, millions of candidates, a few slots, almost no time, is the whole reason recommendation systems look the way they do.
This is the last post in the series, and it’s the clearest example of the funnel the three-systems post was about. RAG had it. Agents had it. But recommendations are the textbook version: cheap wide net, expensive narrow scoring, then the rules a model won’t learn. Let me build the whole thing box by box.
The problem, stated honestly
You can’t score millions of items with a good model on every request. A rich ranking model that looks at this user crossed with this item, with hundreds of features, might take a millisecond per item. Times ten million items, times every user hitting the page, and you’re nowhere near your latency budget or your compute budget.
So the field gave up on scoring everything. Instead it narrows in stages, spending almost nothing per item when there are millions, and spending real compute per item only once there are a few hundred left.
Read that top to bottom and the shape is the point. The first stages are cheap and wide. They’re allowed to be sloppy, as long as they don’t miss the good stuff. The bottom stages are expensive and narrow. They can afford to be careful because there’s almost nothing left to be careful about.
Walk the boxes:
- Request context. Who’s asking, from what device, on what page, at what time. A product page wants “similar and complementary items.” A home feed wants “things you’ll open.” Same machine, different context.
- Feature fetch. A single low-latency read pulls this user’s features (recent activity, long-term profile, current session) from an online store. Milliseconds, or the whole budget is blown before you’ve done anything.
- Candidate generation. The big narrowing. Millions to a few hundred, usually by fusing several sources: a two-tower vector search (more on that below), collaborative filtering (people like you liked this), and plain rules (trending, recent, editorial picks). No single source is enough, so you union them.
- Filtering. Remove what you can’t or shouldn’t show: things they’ve already seen, out-of-stock items, region-locked content, anything blocked. Cheap set operations, big impact.
- Ranking. Now the heavy model earns its keep. On a few hundred candidates it can afford rich user times item cross features, the interactions the retrieval stage literally couldn’t look at, and score each one for the probability you’ll click, watch, or buy.
- Re-ranking. The last pass isn’t about accuracy. It’s diversity so you’re not shown ten near-identical items, freshness, a little exploration, deduping, blending ads, fairness. Business objectives an accuracy model will never optimize on its own.
Then you log it all. Which impressions you served, in what order, with what features. That log, joined later with what the user actually did, becomes the labels you train on. The serving path and the training path are two loops around the same data.
The two-tower trick, which is the whole backbone
The retrieval stage is where the magic lives, and it runs on one idea that’s worth slowing down for: two towers.
You train two separate encoders. One is a user (or query) tower: it takes everything you know about the user and context and produces a vector. The other is an item tower: it takes everything about an item and produces a vector in the same space. You train them jointly so that a user’s vector lands close to the items they’ll actually engage with. Closeness is just a dot product or cosine similarity.
Here’s why that structure is so powerful. Because the two towers are independent, the item tower never needs the user to run. So you compute every item’s embedding offline, in a batch job, and load them all into an approximate-nearest-neighbour (ANN) index. That index can search millions of vectors in single-digit milliseconds.
Then at request time you do almost nothing. You run the user tower once to get a single vector, and you ask the ANN index for its nearest item vectors. That’s how you retrieve a few hundred relevant items out of ten million without breaking a sweat. All the heavy lifting happened last night in a batch job; serving is just one encode plus one lookup.
There’s a catch, and it’s the reason the funnel has more stages after this. The towers can’t see user-times-item interactions. The user vector is computed before it’s ever compared to an item, so the model can’t learn “this user loves this item specifically because of some interaction between them.” That’s the price of precomputability. And it’s exactly why you need a separate ranking stage downstream, one that can look at those cross features, on the small candidate set where you can afford to.
Retrieval is cheap and cares about recall: get the good items into the pool, don’t miss them. Ranking is expensive and cares about precision: order the small pool well. A brilliant ranker can’t rescue an item that retrieval never surfaced, which is why retrieval quality quietly decides your ceiling.
What to precompute, and what to do live
The two-tower split hints at a bigger design principle that runs through the whole system: decide what’s stable enough to precompute, and what’s too volatile to.
Batch / offline
Real-time / online
Stable things go in a batch job because batch compute is cheap and you can afford to redo the world overnight. Volatile things have to be computed live because they didn’t exist a minute ago. The move that makes recommendations feel alive is combining the two: a rich, slowly-built profile of who you are, adjusted at request time by what you’re doing right now. The “you just watched X, so here’s more like X” reaction is that combination in action. Neither half alone would feel as sharp.
The feature store, and the bug it exists to prevent
There’s an unglamorous box in every serious recsys that turns out to matter more than the models: the feature store. And its real job isn’t storage. It’s consistency.
Think about it. Your model trains on features computed in a big offline job over historical data. But at serving time, those same features have to be computed live, in a different codebase, under a latency budget. If the offline “average watch time last 7 days” is computed even slightly differently from the online one, your model sees one thing in training and a different thing in production. That gap has a name: train/serve skew. And it’s the most insidious bug in the whole field, because the model looks fantastic in offline evaluation and quietly falls apart live.
So the feature store defines each feature once and serves it to both worlds: an offline store for training (with point-in-time-correct joins, so you never accidentally leak the future into a training row) and an online store for serving (fast reads). Same definition, both sides. That’s the guarantee you’re paying for. It’s boring, and it’s the difference between a model that works and one that only looked like it did.
Where LLMs and embeddings fold in
Given everything else in this series has been about language models, the honest answer is: they help, but they don’t change the shape.
Content and semantic embeddings power the item tower, and they’re the cleanest fix for cold start. A brand-new item has zero interaction history, so collaborative signals have nothing to work with and it never gets retrieved. But you still have its text, its metadata, its thumbnail. A semantic embedding of that content places the new item near similar existing ones in the shared space, so it can be retrieved on day one. LLMs also generate richer features (summaries, categories, extracted attributes) that feed both towers.
There’s a newer thread too: generative and sequence recommenders that treat a user’s history as a sequence and “generate” the likely next item, the same next-token idea from how LLMs actually work, pointed at item IDs instead of words. Interesting, and real. But the retrieve-then-rank funnel is still the production backbone. LLMs mostly improve the representations inside the boxes, not the arrangement of the boxes. The funnel earns its keep whether the embeddings come from a classic model or a language one.
The cloud services that fill each box
None of this is something you build from raw metal anymore. Every box on the diagram maps to a managed service on each major cloud. Here’s the honest mapping as of mid-2026:
| Box | AWS | GCP | Azure |
|---|---|---|---|
| Feature store | SageMaker Feature Store | Vertex AI Feature Store | Azure ML feature store |
| Embeddings / training | SageMaker | Vertex AI Training | Azure ML |
| ANN retrieval | OpenSearch kNN or Aurora pgvector | Vertex Vector Search (ScaNN) | Azure AI Search vectors |
| Ranking serving | SageMaker endpoints | Vertex AI Prediction | Azure ML online endpoints |
| Low-latency feature cache | ElastiCache / DynamoDB | Memorystore / Bigtable | Azure Cache for Redis / Cosmos DB |
| Streaming events | Kinesis / MSK | Pub/Sub + Dataflow | Event Hubs |
| Offline data / labels | S3 + EMR / Glue | BigQuery + Dataflow | Synapse / Databricks |
The rows are the architecture. Pick a cloud and you’re mostly choosing which brand fills each box, not changing what the boxes do. If you understand the funnel, you can port the design to any of the three by reading down a column.
The gotchas that actually sink these systems
The architecture is the easy part. Here’s what quietly kills recommendation systems in production, in rough order of how often it happens.
1. Train/serve skew and feature leakage
Features computed differently offline vs online, or a feature that secretly encodes the label and won't exist at serving time, gives you a model that's brilliant in evaluation and useless live. Point-in-time-correct joins and a shared feature store are the fix. This is the number one production killer, full stop.
2. Feedback loops and popularity bias
The model recommends what it already surfaces, those items get the clicks, training reinforces them, and the catalog collapses toward a popular few. Rich get richer. Mitigate with exploration (bandits, a dash of randomization), diversity in re-ranking, and logging what could have been shown, not just what was.
3. Cold start and offline/online divergence
New items have no signal and never get retrieved; new users have no profile. Bridge both with content and semantic embeddings plus sensible context defaults. And when a lift in NDCG doesn't move the real business metric, your offline eval isn't measuring reality, so the final arbiter is always an online A/B test.
That last point deserves the final word. You can chase offline metrics, AUC, NDCG, recall@k, forever, and they’re useful for catching regressions cheaply. But they’re a proxy. A model that scores higher offline can lose in production, because offline you’re grading against a world your own past recommendations shaped. The system is a loop, and the only honest measurement of a loop is to run it live against a control. The A/B test is the truth. Everything before it is a hypothesis.
The upshot
A recommendation system feels like mind-reading, and it’s really just discipline about compute. You can’t be smart about millions of items, so you’re cheap and wide first (two-tower retrieval, high recall), then expensive and narrow (a heavy ranker on a few hundred, high precision), then you apply the rules a scoring model would never learn on its own (diversity, freshness, fairness, ads). Cheap recall, expensive precision, policy. The same funnel this whole series has been circling.
That’s the series. Four posts, all built on the same machine: a RAG chatbot, an agent that acts, and this one that ranks. Learn the funnel once and the next system you’re handed is just a matter of asking what’s cheap, what’s expensive, and which rules the model will never figure out for you.