Designing an Agent That Doesn't Go Off the Rails
Here’s a ticket that lands on your desk. “Customer X is unhappy and threatening to churn. Figure out what’s going on and draft a reply.” To actually do that, you’d pull their recent support tickets, check their billing history for a failed charge or a plan downgrade, cross-reference an outage window, form a theory about the likely cause, and only then write something. You don’t know in advance how many of those steps you’ll need, or in what order, until you see what the first ones turn up. If billing looks clean, you go dig in the tickets. If the tickets are quiet, you look harder at billing.
You can’t write that as one prompt. And you can’t write it as a fixed script either, because the path depends on what you find along the way. This is the class of problem where an agent earns its keep: an LLM in a loop, holding some memory, with a set of tools, and the authority to decide what to do next.
This is post three of the AI System Design series. The first post made the case that these systems share one shape; the second worked out a RAG chatbot box by box. Same spine each time: cheap wide recall, expensive narrow precision, then policy. An agent is that spine turned into a loop, and it’s the most powerful and the most dangerous shape of the three. Let me design one end to end, and be honest about when you shouldn’t.
A chain is a straight line. An agent is a loop.
Start with the single most important distinction, because getting it wrong is the number one way these projects go sideways. RAG is a chain: retrieve, then answer. The steps are fixed and always run in that order. An agent loops: it decides, acts, looks at the result, and decides again, and it might go around two times or twelve.
Chain (RAG, a workflow)
Agent (a loop)
The chain is boring, and boring is a compliment. You can unit-test every step, the cost per run is predictable, and it never surprises you at 3am. The agent is the opposite on every axis. Keep that asymmetry in your head for the whole design, because the guiding principle falls right out of it.
The agent loop, up close
When people say “an LLM in a loop,” this is the loop. Six moves, and it goes around until the task is done or a budget stops it.
Notice that plan, guard, and reflect are the three points where model judgment happens, and they map cleanly onto the series funnel: planning is cheap recall (what could I do next), reflecting is expensive precision (was that actually good enough to stop), and guarding is policy (the rules the model won’t enforce on itself). The loop is a funnel that runs over and over.
Use the cheapest thing that works
Before you build any of that, the real question: do you even need an agent? Almost always, you need less than you think. This is a ladder, and you climb it only as far as the problem forces you.
Here’s the discipline that separates production systems from demos: push as much as you possibly can into deterministic workflow, and reserve agentic autonomy for the small part that truly needs it. For our customer-investigation task, “pull the tickets” and “fetch the billing record” are fixed workflow steps. The only genuinely agentic decision is “given what I just found, do I dig deeper into billing or pivot to the tickets?” A workflow graph with a couple of LLM decision nodes beats a fully autonomous “figure it all out” agent in almost every real deployment. It’s more debuggable, cheaper, and it fails in ways you can predict.
Model the graph, don’t let the model wing it
That last point deserves its own section, because it’s a real fork in the design. You can build the agent two ways. One: hand the model all the tools and say “go, decide everything.” Two: model an explicit state graph, nodes are steps, edges are the transitions you allow, and let the model make decisions only at the nodes where a decision is genuinely needed.
The graph version is more work up front and a little less flexible. It’s also enormously more debuggable and testable, because at any moment you know which state you’re in and which transitions are even possible. When something breaks, you can point at the node. A free-form “the LLM decides everything” agent gives you a transcript and a shrug. For production, the trade almost always favors the graph. This is exactly what LangGraph is for, and I wrote a whole post on it, so I won’t relitigate the details here. The point for this design is: prefer the modeled graph.
Tool design is the actual product
An agent is only as good as the tools you give it, and tool design is where most of the quality (and most of the bugs) live. A few principles that matter more than they sound:
- Fewer, well-described tools beat many overlapping ones. The single biggest cause of an agent calling the wrong tool is two tools whose descriptions blur together. If
get_billingandget_invoicessound similar, the model will coin-flip between them. Give each tool one crisp job and one crisp description. - Strict, typed schemas. Every argument typed and constrained. This is your first line of defense against hallucinated arguments, a
datefield that only accepts a real date can’t receive a made-up string. - Validate arguments before you execute. The model will occasionally produce a plausible-looking but wrong argument. Check it in code before the tool runs, and if it’s bad, return a structured error the model can read and recover from, not a stack trace.
- Make tools idempotent. Agents retry. If “issue refund” runs twice because of a retry, you’ve double-refunded. Design write tools so calling them twice with the same request is safe, an idempotency key on every mutating call.
Get this layer right and the loop mostly takes care of itself. Get it wrong, ambiguous descriptions, loose schemas, tools that double-charge on retry, and no amount of clever prompting saves you.
Memory: a small scratchpad and a long shelf
An agent needs to remember things, but “remember” splits into two very different jobs with two very different designs.
What the agent has done and seen during this task: the steps so far, the tool results, the running plan. It lives in the context window.
Facts worth keeping across tasks: this customer's preferences, a resolution that worked before, a learned pattern. Lives in a vector store you retrieve from.
The short-term side is a context-engineering problem (I’ve got a whole post on why AI forgets, if you want the mechanics). The long-term side is closer to the RAG design from post one: it’s a vector store, written to and retrieved from, just with the agent deciding what goes in.
Durable state, so a crash resumes instead of restarts
Our investigation task might be eight steps deep when the process dies. If your agent holds its state only in memory, that crash means starting over: re-pulling every ticket, re-running every tool call, re-paying for every LLM call. Unacceptable.
So persist the state after every step. Write down where you are, what you’ve done, and what you’ve observed, to durable storage, each time around the loop. Now a crash at step eight resumes at step eight. This single decision does double duty: it also enables human-in-the-loop. If you can pause, persist, and later resume, then “wait here for a human to approve” is just a pause state you can sit in for an hour or a day without holding a process open.
Human-in-the-loop, gated by risk
Not every action is equal, and the design should treat them differently. Sort actions by risk tier.
Read-only and reversible actions, look up a ticket, read the billing record, draft (but don’t send) a reply, can run fully autonomously. Nothing bad happens if the agent gets one wrong; you just redo it. But irreversible or costly actions, issuing a refund, sending a message to the customer, deleting a record, moving money, need a human to approve before they fire. And crucially, that approval gate is not an afterthought you bolt on. Design it as a first-class pause state in the graph: the agent reaches the “send refund” node, halts, surfaces its proposed action and its reasoning to a human, and waits. Approve and it proceeds; reject and it re-plans. Because you persisted state, waiting is free.
Observability isn’t optional, and evals score the whole trajectory
You cannot operate an agent you can’t see into. Every loop, every plan, every tool call and its arguments and its result, has to be traced. When an agent does something baffling, and it will, the trace is the only thing that tells you why. This is non-negotiable in a way it simply isn’t for a plain chain, because with a non-deterministic loop, “run it again and watch” doesn’t reliably reproduce the bug.
Evaluation is where agents break the habits you brought from other systems. For a RAG bot you can score the final answer. For an agent, the final answer being right isn’t enough, you have to score the trajectory: did it take sensible steps, call the right tools, in a reasonable order, without a wasteful detour? An agent can stumble to the right answer through five wrong turns, and that’s a system one bad break away from disaster, not a success. And because the loop is non-deterministic, you don’t run each eval case once. You run it many times and track a success rate, because “it worked when I tried it” tells you almost nothing about a system that behaves differently every run.
The cloud boxes, on AWS, GCP, and Azure
Every piece above maps to a managed service on each major cloud. Here’s the whole system, box by box, as of mid-2026.
| What you need | AWS | GCP | Azure |
|---|---|---|---|
| Agent runtime | Bedrock AgentCore | Vertex AI Agent Engine | Foundry Agent Service |
| The reasoner (LLM) | Bedrock (Claude) | Vertex (Gemini) | Foundry (GPT / Claude) |
| Tools / functions | Lambda + API Gateway | Cloud Run or Functions | Azure Functions |
| Durable state | Step Functions + DynamoDB | Workflows + Firestore | Durable Functions + Cosmos DB |
| Long-term memory | OpenSearch or Bedrock KB | Vertex Vector Search | Azure AI Search |
| Tracing | CloudWatch + OpenTelemetry | Cloud Trace | Azure Monitor + App Insights |
On the framework side: LangGraph is the controllable default, an explicit graph with durable checkpoints, which is precisely the “model the graph, persist every step” design above. CrewAI and AutoGen lean into multi-agent role setups (a researcher agent, a writer agent, and so on), useful when a task really decomposes into distinct roles, though multi-agent adds its own coordination headaches, so don’t reach for it by default. And the managed runtimes above will host whichever you pick.
The gotchas that bite in production
Three failure modes show up again and again. None are exotic; all are avoidable with design you put in before launch, not after the incident.
An agent that never decides it's done will happily loop forever, burning money every turn. Guard it with hard budgets: a max step count, a wall-clock timeout, a cost ceiling. Add loop detection (is it calling the same tool with the same args again?), and a forced-termination fallback that returns a graceful "I couldn't complete this" instead of spinning.
The model picks the wrong tool or invents a bad argument, and it's worst when you've given it many similar-sounding tools. Fight it three ways: crisp, non-overlapping tool descriptions; strict argument validation before execution; and limiting the toolset available at each state so the model chooses from a short, relevant menu instead of a sprawling one.
A small mistake early, the wrong customer ID, a misread result, propagates through every later step and comes out as a confidently wrong final answer. This is exactly why tracing is non-negotiable and why evals must score the trajectory, not just the ending. And scope credentials tightly, per tool, least privilege: a bug in an autonomous agent that holds broad write access doesn't make a typo, it does real damage to real data.
The takeaway
An agent is the most capable shape in this series and the one to reach for last. The whole craft is restraint: climb the ladder only as far as the problem forces you, push everything you can into deterministic workflow, and hand the loop only the genuinely dynamic decisions. When you do build the loop, model it as an explicit graph, give it a handful of crisp idempotent tools, split memory into a trimmed scratchpad and a selectively-written store, persist state after every step so a crash resumes and a human can approve the risky moves, and trace and eval the whole trajectory because the thing behaves differently every time you run it.
Do that and you get an agent that can chase down a churning customer across tickets and billing and come back with a grounded draft. Skip it and you get a very expensive random number generator with write access to production. The difference is entirely in the guardrails, and you build them before the launch, not after the postmortem.