blog

Fine-tuning vs RAG vs Prompting: How to Actually Make an AI Model Yours

July 8, 2026 · #ai #rag #fine-tuning #prompting #llm #explainer

Say you want an AI to work for your situation. Maybe it should know your company’s internal docs. Maybe it should always answer in your brand’s voice. Maybe it just keeps getting one specific thing wrong and you want that fixed. You’ve got a real goal, and now you hit the fork that trips up almost everyone: do you prompt it better, bolt on retrieval (RAG), or fine-tune the model?

People get this wrong constantly, and expensively. I’ve watched teams spend months and serious money fine-tuning a model to “know” their data, when a bit of retrieval would have done it in an afternoon, and done it better. I’ve watched others prompt endlessly at a problem that no prompt could ever solve. The three tools look interchangeable from a distance. They are not. Each fixes a different kind of problem, and once you can tell which kind you have, the choice becomes obvious.

This post is the map. If you’ve read my earlier ones on RAG and on prompting methods, this is where they click into a single decision. Let me give you the one mental model that settles it, then the exact signals that point to each tool.

The three levers, quickly

cheapest, first

Prompting Ask better. Use the model's existing knowledge, shape the request with instructions, examples, and format. Nothing about the model changes.

for facts & data

RAG Retrieve relevant documents at question-time and hand them to the model along with the question. It reads your data fresh, every time.

for behavior

Fine-tuning Actually retrain the model's weights on your examples, so a new tendency is baked in permanently. Changes the model itself.

Three different tools for three different jobs. The trap is treating them as "small, medium, large versions of the same thing." They're not a size ladder, they're a toolbox. Using the wrong one doesn't just cost more; it often can't solve your problem at all.

The one mental model that decides everything

Here it is, the single sentence that untangles the whole thing. Tattoo it on your brain:

RAG changes what the model can see. Fine-tuning changes how the model behaves. Prompting works with what it already knows.

RAG

Changes what it SEES. Like handing someone the exact reference document right before you ask them a question. Their knowledge is now current, sourced, swappable.

Fine-tuning

Changes how it BEHAVES. Like sending someone to months of training so a new habit becomes automatic. It's now part of who they are, every time.

Prompting

Works with what it KNOWS. Like asking a knowledgeable colleague a clear, well-framed question. No new info, no retraining, just a better ask.

The whole decision lives here. Got a knowledge or facts problem (it doesn't know your data, or the data keeps changing)? That's a "what it sees" problem, RAG. Got a behavior problem (wrong tone, wrong format, a consistent bad habit)? That's a "how it behaves" problem, fine-tuning. Neither? Just prompt.

This immediately kills the most common mistake. People try to fine-tune a model to teach it facts (“train it on our docs so it knows them”). But facts change, and baking them into weights makes them stale the moment a doc updates, plus the model can’t tell you which document an answer came from. Facts are a “what it sees” problem. That’s RAG’s job, not fine-tuning’s. Fine-tuning is for behavior, not knowledge.

The rule: prompt, then RAG, then fine-tune

Because the three cost wildly different amounts of effort, there’s a natural order. Always climb it in sequence, and stop the moment your problem is solved:

1

Prompt first. It's free and instant. A surprising amount of "we need to fine-tune" turns out to be "we needed a clearer prompt." Try this before anything else.

2

Then RAG. If the gap is that the model doesn't know your data (or the data changes), add retrieval. This is the default for most real production systems in 2026: prompting + RAG.

3

Fine-tune only if neither worked. After prompting and RAG, if there's a specific behavior they still can't fix, then fine-tune, for that behavior, not for facts.

The sequential rule. Each step up costs more time, money, and expertise, so you only climb when the cheaper tool genuinely can't solve it. Most teams never need step 3. The ones who jump straight to it usually regret it.

The decision tree, in four questions

Want it even more concrete? Ask yourself these, in order, and stop at the first “yes”:

Does the model already know this? (general knowledge, common tasks)

yes → just PROMPT. Done.

Is it about facts / data, especially data that changes or needs sources?

yes → RAG. (Never fine-tune for this.)

Is it about consistent style, tone, or a rigid output format that prompting keeps drifting on?

yes → FINE-TUNE (after trying strong few-shot prompting first).

Do you need a small, cheap model to match a big expensive one on one narrow task?

yes → FINE-TUNE (distillation).

Four questions, in order. Notice fine-tuning only wins in two narrow cases: locking in a stubborn behavior, or making a small model punch above its weight on one task. Everything about knowledge routes to RAG or prompting. This is the whole framework on one screen.

Wait, isn’t fine-tuning insanely expensive?

It used to be, and this is why people fear it. Fully retraining even a 7-billion-parameter model means updating every weight, which needs enormous GPU memory, think tens of thousands of dollars for a single training run. That’s the scary version, and for most people it’s overkill.

The thing that changed the game is a technique called LoRA (and its memory-frugal cousin QLoRA). The idea is clever: instead of retraining all the model’s millions of weights, you freeze them and train only a tiny pair of add-on matrices that nudge the model’s behavior. You’re adjusting a small steering attachment, not rebuilding the engine.

Frozen base modelmillions of weights, untouched

+

LoRA adaptertiny, the only part you train

LoRA keeps the giant base model frozen and trains only a small add-on. To give a real sense of scale: on one layer of a 7B model, full fine-tuning adjusts about 45 million parameters; LoRA adjusts around 60 thousand, hundreds of times fewer. You get most of the benefit at a tiny fraction of the cost.

The cost difference is not subtle. It’s the difference between renting a data center and using a gaming GPU:

Full fine-tuning (7B model)~$50,000 of H100 GPUs

QLoRA on the same model~$1,500 gaming GPU (RTX 4090)

Same model, same goal, wildly different bill. LoRA-style methods cut memory needs by 10 to 20 times while keeping roughly 90 to 95% of full fine-tuning's quality. This is why fine-tuning went from "only big labs can afford it" to "a solo developer can do it on their desktop." It's real, and it's accessible now.

Full fine-tuning still wins for the hardest, deepest cases (complex domains, very large training sets). But for most “teach it our house style” jobs, LoRA is the sensible, affordable default.

The best answer is usually “all of the above”

Here’s the grown-up truth: in real production systems, it’s rarely one tool. The strongest setups stack them, each doing the job it’s best at.

A good promptclear instructions + format

wrapped around RAGfresh facts, cited from your documents

running on a lightly fine-tuned modellocked-in house voice + cheaper to run

The hybrid pattern most mature systems land on: RAG supplies the current, sourced facts; a light fine-tune supplies the consistent voice and format; a clear prompt ties it together. Each layer does what it's uniquely good at. Facts from retrieval, behavior from fine-tuning, framing from prompting.

The full comparison

	Prompting	RAG	Fine-tuning
Changes	Nothing (just the ask)	What the model sees	How the model behaves
Best for	What it already knows	Facts, your data, fresh info	Style, tone, format, narrow skills
Freshness	Frozen at training	Instant, swap a document	Goes stale, needs retraining
Cite sources?	No	Yes, points at the document	No, weights are opaque
Cost / effort	Lowest	Medium (build + iterate)	Highest (data + training)
Try it...	First	Second	Last, only if needed

The whole decision on one card. The two rows that resolve most arguments: "cite sources?" (only RAG can, which is why audited/regulated use cases need it) and "freshness" (fine-tuning quietly rots as your data moves on). When in doubt, these two columns usually point the way.

The takeaway

Stop asking “which is best.” None is best; each is best at its own job. The question that actually matters is: what kind of problem do I have?

If the model just needs to be asked better, that’s prompting. If it needs to see information it doesn’t have, especially changing information you need to trust and source, that’s RAG. If it needs to behave differently in a stable, repeatable way, tone, format, a narrow skill, that’s fine-tuning, and thanks to LoRA it’s finally affordable. And when you build something real, you’ll probably reach for all three, each doing the one thing it does best.

The teams that ship great AI aren’t the ones who fine-tune the most. They’re the ones who correctly diagnose which lever the problem actually needs, and reach for the cheapest one that solves it. Prompt first. RAG next. Fine-tune last, and only on purpose. That order will save you more time and money than any single technique ever could.

← Back to blog