Building an Agent to Sift Years of Messy Data Into a Plan
Here’s a problem that lands on a lot of engineers’ desks, in one shape or another. A client shows up with years of data: advertising campaigns, spend records, creative performance, audience reports, spreadsheets, exports, dashboards nobody has opened since 2022. It’s a mountain. And the ask is deceptively simple: “Go through all of it, figure out what actually matters, throw away the noise, and give us a plan.”
Most of that data is noise. Genuinely. A dead campaign from three years ago, a duplicated export, a metric nobody acts on. The signal, the stuff that should shape the next plan, is a small fraction buried in the pile.
I want to walk through how I’d actually build an agentic tool for this. Not the buzzword version, the real engineering decisions. And the interesting part is the fork in the road early on: there are three obvious ways to approach it, the most tempting one is a trap, and understanding why it’s a trap is what leads you to the design worth shipping. Let me think it through out loud.
First, what does “good” even look like?
Before choosing an approach, pin down what success means, because it shapes everything:
- Relevant, not just recent. A great campaign from two years ago might matter more than a mediocre one from last month. “Relevant” is a judgment call, not a date filter.
- Traceable. When the plan says “double down on video,” the client will ask why. Every conclusion must point back to the data that justified it. No unsourced claims.
- Affordable and repeatable. Years of data is a lot of tokens. If the tool costs a fortune or takes an hour per run, it won’t get used.
- Honest about gaps. If the data doesn’t support a recommendation, the tool should say so, not invent one.
Hold those four in mind. They’re the scorecard we’ll judge each approach against.
The three approaches on the table
Modern models have huge context windows. So just feed everything in and ask for a plan. Tempting because it's the least code.
Index everything, and when you need a plan, retrieve the most relevant chunks and reason over just those. Cheap and focused.
An agent that retrieves in passes, judges what's relevant, discards the rest, and only then plans, checking its own work as it goes.
Approach 1: just dump it all in. Why it’s a trap.
This is where most people start, and it’s seductive. Context windows are enormous now, so why not pour in years of data and let the model sort it out? One prompt, no pipeline, done.
Here’s why it falls apart, and it’s not just cost.
The “lost in the middle” problem. This is the killer. Models don’t read a giant context evenly. They pay strong attention to the start and end of what you give them, and they get noticeably worse at using information stuck in the middle. Research puts real numbers on it: when the fact you need sits in the middle of a long context, accuracy can drop by 15 to 20% versus when it’s near the top.
And the cost is brutal. You pay per token, every single run. Stuffing years of data into every query is like rebuilding the whole library each time you want to look up one fact. Compared to retrieving only what’s relevant, the full-dump approach can cost roughly 10 to 15 times more per query and take far longer. Against our scorecard: it’s not reliable (misses the middle), not cheap, and not repeatable. It fails three of four. Scratch it.
The honest exception: if the customer’s data were small, say it comfortably fit in a prompt with room to spare, dumping it in is genuinely fine and you shouldn’t over-engineer. But “years of ad data” is not that. This is the wrong tool for this size.
Approach 2: pure RAG. Better, but it can’t think in steps.
So if you can’t feed everything in, feed in only what’s relevant. That’s Retrieval-Augmented Generation, RAG. You index all the data once (turning it into searchable vectors by meaning, using embeddings), and when you need a plan, you retrieve just the most relevant pieces and reason over those.
This is a big leap forward and hits most of our criteria: it’s cheap (retrieving relevant chunks cuts token use by around 90% versus the full dump), it’s scalable, and because every answer is tied to retrieved documents, it’s traceable. Good.
But here’s where plain RAG runs out of road for this task. RAG does one retrieval, then answers. Our problem isn’t a single-answer question, it’s a judgment-and-planning job with several phases:
- First decide what topics even matter across years of data.
- Then, for each, dig deeper and separate signal from noise.
- Then judge whether the evidence is strong enough to act on.
- Then synthesize all of that into a coherent plan.
One retrieval pass can’t do that. It grabs some chunks and produces some text, but it can’t loop, can’t say “these results are thin, let me search differently,” can’t first filter and then plan. It has no agency. It retrieves; it doesn’t reason across steps.
Plain RAG is the right engine. It’s just missing a driver.
Approach 3: the agentic pipeline. What I’d actually ship.
Here’s the design. Take RAG as the engine, and put an agent in the driver’s seat, something that can act in steps, judge results, and loop until the job’s done. (If “agent” is fuzzy for you, it’s an AI that directs its own process toward a goal, deciding its next move as it goes.) The whole thing becomes a funnel: start with the mountain, filter hard, and only plan on what survives.
The magic is stage 3. That’s where “agentic” earns its name. A plain pipeline would retrieve once and hope. This agent evaluates what it found, notices when results are weak, and retrieves again with a better query, the same way a good analyst circles back when the data looks thin. It filters before it plans, so the planning step never drowns.
Watch what the funnel does to the volume, which is the whole point:
Why this one wins, measured against the scorecard
Let’s be disciplined and score all three against the four criteria we set at the start. This is the actual decision, not a vibe.
| Criterion | Dump it all | Pure RAG | Agentic pipeline |
|---|---|---|---|
| Relevant, not just recent | Misses the middle | Retrieves relevant, but one-shot | Judges relevance in passes ✓ |
| Traceable to sources | Hard, it's a blob | Yes ✓ | Yes, per finding ✓ |
| Affordable / repeatable | No, 10x+ cost | Very ✓ | Costs more than RAG, still fine ✓ |
| Honest about gaps | Tends to hallucinate | Better | Can flag thin evidence ✓ |
| Multi-step judgment | No | No | Yes, the whole point ✓ |
The honest cost of the choice
I’m not going to pretend the agentic approach is free. It’s the right call here, but it comes with real trade-offs, and a good engineer names them:
There are also the unglamorous realities you’d plan for: guardrails so the agent can’t loop forever running up a bill (a hard cap on retrieval passes), a human check on the final plan before it reaches the client (it’s a draft from a sharp analyst, not gospel), and observability so when the client asks “why did it recommend this?”, you can show the exact path from data to conclusion. Those aren’t optional extras; they’re what makes it trustworthy enough to actually deploy.
The takeaway
The instinct with a mountain of data is to throw the whole mountain at a big model and hope. That’s the one approach almost guaranteed to disappoint: expensive, and blind to whatever’s buried in the middle. The better instinct is a funnel. Index everything so it’s findable, then put an agent in charge of retrieving in passes, judging what’s relevant, discarding the noise, and only then planning over the clean set that survives.
And the real lesson underneath the ad-data example is a way of thinking, not a recipe. When you face any “make sense of a huge pile” problem, don’t ask “which fancy technique should I use.” Ask “what does good actually look like here,” write down the criteria, and let the task pick the approach. Sometimes that’s a humble one-shot RAG. Sometimes, when real judgment across many steps is needed, it’s an agent. The skill isn’t reaching for the most powerful tool. It’s reaching for the one the problem actually calls for, and being able to say exactly why.