Fine-Tuning vs Prompting vs Retrieval: Which Lever to Pull First

When a model does not do what a business needs, teams reach for one of three fixes: write better instructions, give it access to the right information, or train it further on examples. The three are often discussed as if they were competing options. They are better understood as solutions to different problems.

Choosing the wrong one is expensive. Fine-tuning a model to teach it facts that change weekly wastes effort. Stuffing instructions into a prompt to fix a deep behavioural issue never quite works. Knowing which problem you actually have is most of the decision.

Start with what is actually wrong

Before choosing a technique, name the failure. There are really three kinds.

The model does not know something it needs to: your product details, your policies, this customer’s order history. That is a knowledge problem.

The model knows enough but behaves wrongly: it uses the wrong tone, ignores the format you need, or handles a task in a way that does not match your expectations. That is a behaviour problem.

The model is capable but too slow, too expensive, or too inconsistent at scale on a narrow task. That is an efficiency problem.

Each points to a different lever.

Prompting: the cheapest lever, and the first one to pull

Prompting means giving the model better instructions and examples. It is the fastest to try, costs almost nothing, and can be changed in minutes. It should always be the first attempt.

A good prompt specifies the role, the task, the constraints, the output format, and a few examples of what good looks like. Surprisingly many problems that look like they need training turn out to be an under-specified instruction.

Its limits are real. A prompt has finite space, and everything in it is paid for on every call. It can nudge behaviour but it cannot reliably teach a model a large body of information or a deeply specialised skill. And long, intricate prompts become fragile, where changing one line shifts behaviour elsewhere in ways that need testing, which is why the measurement in evaluating an AI agent before you trust it matters so much here.

Retrieval: for knowledge that changes or is private

Retrieval means fetching relevant information at the moment of the question and placing it in front of the model. It is the right tool when the model needs facts it was never trained on: your documentation, your data, anything current.

Its great advantage is that knowledge stays outside the model, where it can be updated instantly, controlled, and cited. Change a policy document and the answers change. Remove a record and it is gone. You can also show where an answer came from, which builds trust, as discussed in retrieval-augmented search for business.

Retrieval has its own difficulties. Quality depends on finding the right passages, which is harder than it sounds, and the corpus has to be kept fresh. But for a knowledge problem it is almost always better than the alternative of trying to bake facts into the model.

Fine-tuning: for consistent behaviour at scale

Fine-tuning means training a model further on your own examples so that it internalises a style, a format, or a narrow skill. It is the heaviest lever: it needs good training data, effort to run and evaluate, and it produces a model you must maintain as the base models improve.

It is a good fit in particular cases. A very consistent output style or format that prompting cannot hold. A narrow, high-volume task where a smaller, cheaper fine-tuned model can match a larger one, cutting cost and latency. A specialised skill or vocabulary that the general model handles poorly.

It is a poor fit for teaching facts. Fine-tuning does not reliably make a model remember specific information, and anything it does learn goes stale, with no good way to update or cite it. Using it for knowledge is the most common misuse.

Compare them honestly

Prompting is fast to iterate, cheap, and reversible, but limited in what it can change. Retrieval keeps knowledge current and citable, and adds a system to build and maintain. Fine-tuning can deliver consistency and efficiency at scale, with the highest upfront cost and the slowest feedback loop.

The costs also differ in kind. Prompting and retrieval costs are paid on each call and in engineering of the surrounding system. Fine-tuning costs are paid mostly up front and in upkeep, with potentially lower per-call costs afterwards. That last point links to the spending discipline in AI agent cost control: at high volume on a narrow task, a smaller tuned model can be a real saving.

They combine well

Most production systems use more than one. A prompt sets the behaviour and format, retrieval supplies the facts, and in some cases a fine-tuned model handles a narrow step cheaply. The order in which you add them matters: prompt first, retrieval next if knowledge is the gap, fine-tuning last and only with evidence that the first two are not enough.

Structured output requirements, covered in generating into a schema, are another example of something you can often solve by prompting and validation before reaching for training.

Measure before and after

Whichever lever you pull, test it against a fixed set of real examples, and compare. Without that, you cannot tell whether a change helped, and fine-tuning in particular can quietly make some behaviours worse while improving others. Build the evaluation set before you start, and keep it separate from any training data.

A practical order of operations

Try a clear prompt with examples and see how far it gets. If the gap is knowledge, add retrieval. If the gap persists and is behavioural or a matter of cost at scale, and you have enough good examples, consider fine-tuning. Each step is more expensive and slower to undo than the last, so move to the next only when the evidence says the previous one has run out.

We help teams choose the simplest approach that works. See how we approach agent work, or read about scoping an AI MVP.