Fine-tuning or retrieval (RAG)? How to choose.

Whether to use fine-tuning or retrieval comes down to one distinction: fine-tuning changes how a model behaves, and retrieval changes what it knows at the moment it answers. If the problem is facts that change, such as prices, policies or contract terms, you need retrieval-augmented generation (RAG). If the problem is a habit, such as a format, a tone or a set of categories the model keeps getting wrong, fine-tuning can help, once clear prompting has genuinely failed.

The question usually arrives in the first workshop of a project, phrased as a budget decision. Do we train a model on our data, or build search on top of one? It gets answered quickly, and it gets answered backwards more often than not.

This guide explains what each option actually does, how a retrieval pipeline works and fails, what fine-tuning really needs, and the questions that settle the choice. It finishes with how to evaluate whichever route you take and where to start.

Prompting, retrieval and fine-tuning in plain terms

There are three ways to shape what a language model does for your business, and they work at different levels. Prompting changes the instructions for a single call. Retrieval changes the information the model is given. Fine-tuning changes the model itself.

What is retrieval-augmented generation?

Retrieval-augmented generation (RAG) is a pattern where the system searches your own documents for passages relevant to a question and puts them into the prompt, so the model answers from that material. The model itself is unchanged. What changes is what it has in front of it when it writes.

The practical consequence is that knowledge lives outside the model, in documents you control. Update a policy, re-index it, and the next answer reflects the change. Because the system knows which passages it supplied, it can also show the user where an answer came from.

What is fine-tuning?

Fine-tuning is further training of an existing model on example inputs paired with the exact outputs you want, so that it produces that kind of output more consistently. The result is a new version of the model that you call instead of the original. Parameter-efficient methods update only a small set of weights rather than the whole model, which makes training lighter, but the principle is the same.

Fine-tuning teaches patterns: structure, vocabulary, tone, how to classify, when to decline. Whether you can fine-tune a particular model at all, and on what terms, depends on the provider, or on whether you run an open model on your own infrastructure.

What can prompting alone do?

Prompting alone can handle many of the problems that get put forward as fine-tuning candidates. A clear instruction, a few worked examples inside the prompt and a defined output structure change a model's behaviour immediately, with nothing to train or host.

Many providers also let you require output that matches a schema you define, which removes a whole class of formatting failures without any training. Prompting cannot give the model facts it has not been shown, and very long prompts tend to be slower and cost more, but as a first step it is hard to beat.

What is the difference between fine-tuning and RAG?

The difference is that fine-tuning adjusts the model's habits, while RAG controls the information in front of the model when it answers. Sort your requirement into one of those two buckets and the right approach usually becomes clear.

Knowledge problems

A knowledge problem is one where the model needs a fact it cannot be expected to hold: this customer's contract, last quarter's figures, the current version of the returns policy. These call for retrieval. The facts change, and a model trained last month has last month's version built in, with no way to tell you so.

The tell for a knowledge problem is that the correct answer changes without the question changing. Ask what the notice period is in March and again in June, and the right answer may differ because someone edited a document.

If updating a spreadsheet should change what the system says, no amount of training will keep up.

Behaviour problems

A behaviour problem is one where the model needs to respond in a particular shape every time: a fixed set of fields, a label from your own categories, a tone your compliance team signed off, a consistent way of declining. These are candidates for fine-tuning, once prompting has been properly tried.

The tell for a behaviour problem is that you can describe the failure without referring to any fact: it is too wordy, it keeps hedging, it returns six fields when you asked for four. Those are habits, and habits are what training adjusts.

Does fine-tuning on your documents teach the model facts?

Not reliably. Fine-tuning on a pile of company documents mostly shifts how the model writes, and any facts it does absorb come back inconsistently, with no source attached and no easy way to correct them.

The usual reasoning goes: our domain is specialised, generic models do not understand it, so we must train one on our documents. It sounds right. It is also an expensive way to discover that the model understood the domain perfectly well and simply did not have the relevant document in front of it.

After training on raw documents, ask a specific question and the model may well answer in your house style, with your vocabulary, and sometimes with the wrong number, delivered confidently. It has learned the register of your documents more than their content. The wrong answer now sounds exactly like the right ones, which makes it harder to catch.

There are structural problems too. You cannot ask a trained model which document a fact came from. You cannot remove one outdated fact without training again. And you cannot restrict who sees what, because anything the model absorbed is available to everyone who uses it.

How retrieval works and where it fails

A retrieval pipeline has a preparation stage that runs whenever documents change, and a query stage that runs every time someone asks a question. Neither stage is exotic, but each involves choices that determine whether the answers are any good.

Splitting documents into chunks

Documents are first converted to clean text and split into chunks: passages small enough to search and to pass to a model. A chunk might be a clause, a section under a heading, or a few paragraphs. The aim is that each chunk makes sense on its own and carries enough context to be useful.

Good chunking follows the document's structure rather than cutting every few hundred words regardless of meaning. It also attaches metadata to each chunk: the source document, the section, the date or version, and who is allowed to see it. If the sources are scanned PDFs, text quality matters before any of this, which is why extraction that copes with messy real-world scans is often the first piece of work.

Embeddings and hybrid search

Each chunk is turned into an embedding, a list of numbers that represents its meaning, so that passages about similar things end up close together. When a question arrives, it is embedded the same way, and vector search finds the chunks whose meaning is closest to it.

Vector search is good at matching meaning when the wording differs, but it can miss exact terms such as product codes, clause numbers and names. Keyword search is better at those. Many production systems run both and merge the results, which is called hybrid search, and filter on metadata such as date or product so that a question about the current policy does not pull last year's version.

Reranking and citations

Search returns candidates, usually more than the model needs. A reranker, a model built to score how well a passage answers a specific question, reorders them so that only the strongest few go into the prompt. It adds a step, and it tends to improve which passages reach the model.

The prompt then tells the model to answer only from the supplied passages, to cite the passage behind each claim, and to say plainly when the passages do not settle the question. Because every passage has an identifier, the system can check that each citation points to something it actually retrieved, and show the user the source.

Where retrieval goes wrong

When a RAG system gives a bad answer, check retrieval before blaming the model. The common failures are predictable.

• Poor chunking. A clause split from its heading, or an exception separated from the rule it modifies, gives the model half the picture, and it answers confidently from half.
• Stale indexes. A document was updated but not re-indexed, or a withdrawn document was never removed, so the system cites a version that no longer applies.
• Ignoring permissions. If access rules are not applied at search time, the assistant can end up quoting a salary review to someone who should never see it.
• Retrieving the wrong passage. Similar documents, such as the terms for two products or two regions, look alike to search, and the model answers from the wrong one.

Each of these has an ordinary engineering fix: structure-aware chunking, re-indexing triggered by document changes, permission filters applied before results reach the model, and metadata filters plus reranking for look-alike content. None of them is fixed by training the model.

What fine-tuning needs and what it is good for

Fine-tuning needs labelled examples, an evaluation set and a plan for retraining. Each of those usually takes more work than the training run itself.

Labelled examples, not documents

Training data for fine-tuning is a set of examples: a realistic input and the exact output you want for it. Teams often have thousands of documents and nowhere near enough labelled pairs. Producing them is the real cost, because someone who knows the work has to write or check every output.

The examples need to be consistent, since the model will learn your inconsistencies as faithfully as your rules. They also need to cover the variety of real inputs, including the awkward ones. Providers publish their own guidance on formats and minimum sizes, and it varies.

An evaluation set and a retraining plan

You also need a separate set of examples the model never trains on, so you can measure whether the fine-tuned version beats a well-prompted baseline. Without that comparison, nobody can say whether the training helped, did nothing, or made things worse outside the examples it saw.

Then there is maintenance. When the behaviour you want changes, such as new categories or a revised tone, you update the examples and train again. When a provider retires the base model, you generally have to repeat the training on its replacement.

When is fine-tuning worth it?

Fine-tuning is worth it when you need a stable behaviour that prompting cannot hold consistently, and you have examples that demonstrate it. Typical good fits:

• Fixed formats. Output that must follow a precise structure or house convention every time, beyond what schema enforcement already guarantees.
• Tone and style. A voice that is hard to describe in instructions but easy to recognise in examples.
• Classification. Sorting inputs into your own label set, especially where the boundaries between labels are subtle.
• Refusal behaviour. Declining consistently when a request is out of scope or the evidence is missing.
• Narrow tasks on smaller models. Getting a smaller model to do one well-defined job reliably, which can shorten prompts and reduce running cost.

Why you should try prompting first

You should try prompting first because it is fast, cheap to change, and gives you the baseline every later decision depends on. Improving a prompt can be a morning's work, while fine-tuning means collecting examples, training, evaluating and deploying a new model version.

Work through the prompt in order before concluding that it has failed:

1. Write explicit instructions. State the task, the audience, the constraints and what to do when information is missing. Vague prompts produce vague behaviour.
2. Add worked examples. Include a few realistic inputs with the exact outputs you want, including at least one awkward case. Models tend to follow examples closely.
3. Constrain the output. Where the provider supports it, require a defined structure so that formatting stops being a matter of luck.
4. Test on real inputs. Run the prompt against a set of genuine cases, not the handful that came to mind, and write down exactly how it fails.

If the failures that remain are behavioural, consistent and describable, you now have a real case for fine-tuning, plus the test cases to prove whether it worked. If they involve missing or outdated facts, the fix is to supply those facts at answer time, which is retrieval.

The questions that settle it

A few questions, asked before anything is built, usually close the argument quickly.

• How often does the right answer change? If it changes daily or weekly, you need retrieval, because retraining at that pace is impractical and each trained version would soon be out of date.
• Must answers cite their sources? If users or auditors need to see the clause behind an answer, retrieval gives you that directly. A fine-tuned model has no source to point at.
• Do labelled examples exist? That means inputs paired with the exact outputs you want, checked by someone who knows the work. If nobody can produce them, fine-tuning is premature rather than wrong.
• Who is allowed to see what? If different users have different access, that has to be enforced when passages are retrieved, which is something training cannot do.
• What latency and running cost can you accept? Retrieval adds a search step and longer prompts to every call. Fine-tuning adds training and retraining, and may allow shorter prompts or a smaller model. How that balances out depends on your provider and volume, and keeping model costs down covers the levers in more detail.

The table below puts the three approaches side by side.

Which approach fits common use cases?

Most use cases map cleanly to one approach once you ask whether the problem is facts or behaviour. The table gives a sensible starting point for common requests, and your own evaluation should have the final word.

Retrieval does not always mean searching documents. For structured data that changes constantly, such as stock levels or order status, the right form of retrieval is usually a direct query to the system of record, with the result passed to the model. Vector search suits text; a database query suits a stock count.

When you need both

Mature systems often use both, for different jobs, with a clear boundary between them. Retrieval supplies the facts. A fine-tuned model, or more often a carefully prompted one, decides the shape of the response.

A document assistant is the clearest example. Retrieval finds the clauses that matter to the question. The model's job is then narrow and stable: answer only from what it was given, cite the clause, and say plainly when the clauses do not settle the question. That last behaviour can be worth training, because models are often weak at it by default and it is the one users judge you on.

retrieval:  which clauses are relevant   # changes daily
model:      how to answer from them      # changes rarely

Keeping those two lines separate is most of the architecture. It also tells you where to look when something goes wrong: a wrong fact is a retrieval bug, a wrong shape is a model bug, and you can fix either without touching the other.

The order of work matters too. In projects like these, retrieval and prompting come first, and fine-tuning is added only for the behaviour that remains stubborn once the rest is measured. That is how we approach assistants and chat systems built on your own documents.

How do you evaluate the approach you chose?

You evaluate it with a fixed set of real questions and known good answers, scored the same way before and after every change. The details differ by approach, but the principle does not: decide what correct looks like before you build, then measure against it.

Evaluating retrieval

Test retrieval on its own before judging answers. For each question in the set, record which passages should be found, then check whether they appear in the top results. If the right passage is not retrieved, no model can answer correctly, and rewriting the prompt is wasted effort.

Then score the answers with short yes or no checks: does it address the question, is every claim supported by the supplied passages, do the citations point to the right source, and does it say so when the passages are insufficient. Narrow checks like these are easier to score consistently than a general quality rating.

Evaluating a fine-tuned model

Compare a fine-tuned model with the prompted baseline on held-out examples it never trained on. Check the target behaviour, and if the model does anything else in your system, check a sample of those tasks too, since narrow training can make a model worse at things outside its examples.

Keep the evaluation set growing. Every escalation, complaint or corrected answer from production becomes a new case, which stops the suite passing forever while quality drifts. The guide to testing AI systems whose output varies from run to run covers how to write checks that survive the model choosing different words.

Common mistakes to avoid

These mistakes come up repeatedly, and most of them are decisions made before anyone has measured anything.

• Fine-tuning to add knowledge. Training on a folder of documents to make a model know your facts produces confident answers you cannot trace or update.
• Skipping the baseline. Without a well-prompted version to compare against, nobody can say whether a fine-tune earned its cost.
• Indexing once. A retrieval index built for the demo and never refreshed drifts away from the documents it claims to represent.
• Leaving permissions for later. Access control added after launch is hard to get right, because the index was never designed to carry it.
• Blaming the model for retrieval bugs. Rewriting prompts when the wrong passage was retrieved treats the symptom and hides the cause.
• Judging by demo questions. A few hand-picked questions say little about real traffic, which is one of the reasons AI pilots stall before they reach production.

Key takeaways

• Fine-tuning changes how a model behaves, and retrieval changes what it knows when it answers.
• If the right answer should change when a document changes, the problem needs retrieval rather than training.
• Fine-tuning on raw documents does not make facts reliable, traceable or easy to correct.
• Good retrieval depends on sensible chunking, hybrid search, reranking, fresh indexes and permission filters.
• Fine-tuning needs labelled input and output examples, a held-out evaluation set and a plan for retraining.
• Try clear instructions, worked examples and structured output before training anything.
• Strong systems often combine both, with retrieval for facts and a prompted or fine-tuned model for the shape of the answer.
• Whatever you choose, measure it against a fixed evaluation set that grows from real failures.

Where to start with your own project

Start by writing down, in plain words, what the system must do and what it gets wrong today. If that description mentions facts, documents or anything that changes, begin with retrieval and a good prompt. If it describes a habit, begin with better prompting and an evaluation set, and keep fine-tuning in reserve.

Retrieval with a clear prompt is usually quicker to stand up for knowledge-heavy work, and it can be corrected by editing a document instead of retraining. Reach for fine-tuning when you have a behaviour you can describe precisely, examples that demonstrate it, and evidence that prompting did not hold it.

If you want a second opinion before committing budget either way, our AI strategy and review work looks at exactly this kind of decision. Or schedule a call and we can talk through your use case, your documents and the behaviour you need.

Frequently asked questions

Is RAG cheaper than fine-tuning?

Often at the start, but it depends on your volume and provider. RAG needs no training run and is updated by editing documents, so the upfront cost is usually lower. Each call does carry a search step and a longer prompt. Fine-tuning adds the cost of labelling examples, training and retraining, but can allow shorter prompts or a smaller model. Estimate both against your expected traffic before deciding.

Does RAG stop a model from making things up?

It reduces the problem but does not remove it. Giving the model the right passages and instructing it to answer only from them makes invented answers less likely, and citations let people check the source. A model can still misread a passage, blend two sources or answer when the passages are silent. Citation checks, clear instructions to admit gaps and regular evaluation keep this under control.

How many examples do I need to fine-tune a model?

There is no single number, because it depends on the task, the model and the provider. A narrow, consistent task needs fewer examples than a subtle one with many edge cases. What matters most is that the examples are correct, consistent and representative of real inputs. Start with a modest set, measure against a held-out evaluation set, and add examples where the model still fails.

Do long context windows make RAG unnecessary?

Not usually. Some models accept very long inputs, so a small document set can sometimes be sent in full. For larger or changing collections, sending everything on every call tends to be slower and more expensive, and the model may overlook the one passage that matters. Retrieval also lets you enforce permissions and cite sources, which a long prompt of everything does not do on its own.

How often should a RAG index be updated?

As often as the source documents change. A sound approach is to re-index a document automatically when it is created, edited or deleted, rather than rebuilding everything on a fixed schedule. Withdrawn documents must be removed from the index, not just superseded, or the system may still cite them. For fast-changing structured data such as prices or stock, query the live system at answer time instead.

Does fine-tuning make a model more accurate?

It can make a model more consistent at a specific behaviour, which looks like higher accuracy on tasks such as classification or formatting. It does not make the model reliably more accurate about facts, and narrow training can make it worse at tasks outside its examples. The only way to know is to compare the fine-tuned model with a well-prompted baseline on the same held-out test set.

Can a RAG system respect document permissions?

Yes, if permissions are designed in from the start. Each chunk carries the access rules of its source document, and every search is filtered by what the current user is allowed to see before any results reach the model. Those rules must stay in sync with the source systems. Hiding restricted content through prompt instructions alone is not a reliable control.

Schedule a call