RAG vs fine-tuning: how do you make an AI know your business?
This is the decision behind almost every "can we use AI on our own data" conversation, and it is routinely answered backwards. The short version: retrieval teaches a model facts, fine-tuning teaches it behaviour, and most businesses asking the question have a facts problem. We build both and recommend retrieval far more often.
Use retrieval-augmented generation when the model needs to know things — your documents, policies, products, prices — because the knowledge stays outside the model, updates the moment the source does, and can cite where an answer came from. Use fine-tuning when the model needs to behave a particular way: a house format, a specialist tone, a rigid output structure. Facts are a retrieval problem; style is a fine-tuning problem, and confusing the two is the most expensive mistake in this area.
When each one is the right call
Use RAG (retrieval)
- The model needs to know facts that live in your documents
- That information changes — prices, policies, stock, availability
- Answers must cite a source, or be traceable to one
- Different users are allowed to see different things
- You want to correct a wrong answer by editing a document
Use fine-tuning
- You need a consistent house voice or a rigid output format
- The task is narrow, repetitive and specialised
- You have hundreds of good, consistent examples of the desired output
- Prompting gets you close but drifts over long or varied inputs
- You want a smaller, cheaper model to punch above its weight on one job
The differences that actually matter
| RAG | Fine-tuning | |
|---|---|---|
| What it changes | What the model can see at the moment it answers | How the model behaves, permanently |
| Best at | Facts, documents, anything that changes | Tone, format, style, narrow specialist tasks |
| Updating it | Edit the source document — live immediately | Rebuild the dataset and retrain |
| Citations | Natural — you know which passage was used | Not possible; the knowledge is baked in and unattributable |
| Per-user permissions | Straightforward — filter what can be retrieved | Not possible; the model knows what it knows for everyone |
| What you need to start | Your documents, and a way to keep them current | Hundreds of consistent, high-quality example pairs |
| Getting a wrong answer fixed | Edit the document, or fix the retrieval | Retrain, and hope you did not break something else |
| Cost shape | Ongoing: retrieval plus a larger prompt on every call | Upfront training, then cheaper calls |
| Sensitive data | Stays in your store; retrieved per request | Absorbed into model weights |
| Typical failure | The right passage was never retrieved | Confident, fluent, out of date |
Neither column is a verdict. Every row is a trade-off, and which side of it you want depends on the project — which is what the sections below work through.
RAG projects fail at retrieval, not at generation
When a retrieval system gives a bad answer, the instinct is to blame the model and reach for a bigger one. In our experience that is almost never the problem. The model answered perfectly well using the passages it was given; the passages were wrong.
Retrieval quality is decided by unglamorous work done before any model is involved. How documents are split matters enormously — chunks that cut a policy in half produce answers that are half a policy. Tables, PDFs and slide decks need extracting properly rather than flattened into noise. Near-duplicate documents with different dates need resolving, because the system has no way to know which one you meant. And a pure similarity search will happily return five passages that are all about the right topic and none of which contain the answer.
The fixes are equally unglamorous: better chunking with real overlap, combining keyword search with vector search so exact terms are not lost, reranking the candidates before they reach the model, and attaching metadata so the system can prefer the current version of a document over last year's.
This is why we measure retrieval separately from answers. If you cannot say what proportion of questions retrieved the right passage at all, you cannot tell whether a bad answer is a knowledge problem or a model problem — and you will spend money on the wrong one.
Knowing and behaving are different problems
A useful way to think about it: fine-tuning changes the employee, retrieval changes the filing cabinet they have access to.
If your problem is that the model does not know your refund policy, your product catalogue, your prices or what you told a specific customer last March, that is a filing cabinet problem. Baking those facts into weights is the wrong tool in three separate ways: they go stale the moment anything changes, you cannot show where an answer came from, and you cannot stop one user seeing another user's information because the model simply knows it.
If your problem is that the model writes in a register that is not yours, ignores a required structure, or fails to follow a specialist convention that would take a page of instructions to describe, that is an employee problem, and training is the right shape of solution.
The reason this gets confused is that both are described as "training the AI on our data", which flattens the distinction. The question worth asking before any build is: when this gets an answer wrong, what would I want to change to fix it? If the answer is "a document", you want retrieval. If the answer is "how it writes", you want fine-tuning.
Try the cheap things first, and in this order
There is a reliable ladder here, and most projects stop before the top rung. Skipping straight to fine-tuning is the most common and most expensive mistake we see.
Start with the prompt. A clear instruction, an explicit output format and two or three good examples in the prompt itself solve a surprising share of behaviour problems for the cost of an afternoon, and unlike training they can be changed in minutes.
Then add retrieval, if the gap is knowledge. This is where the majority of business use cases end, because the majority of business use cases are questions about material the business already has.
Then improve retrieval rather than the model — better chunking, hybrid search, reranking, metadata filters. This is where the real quality gains live, and it is almost always cheaper than the alternatives.
Only then consider fine-tuning, and only for the specific behaviour that survived all of the above. At that point you know exactly what problem you are solving, you have real examples of correct output collected from the working system, and you have an evaluation set to prove the training helped rather than just changed things.
They are not alternatives — most mature systems use both
Framing this as a versus is convenient for a comparison page and slightly misleading in practice. The two operate on different axes, so a system can and often should do both.
A realistic mature setup looks like this: retrieval supplies the facts, so answers are current and citable; the prompt supplies the rules of engagement, including what to do when the retrieved material does not contain the answer; and a fine-tuned model handles the specific behaviour that prompting could not pin down, such as a strict output schema or a specialist register.
Crucially, fine-tuning in that setup does not carry the knowledge. That separation is what keeps the system maintainable — you can update every fact in the product without touching the model, and you can change the model without re-teaching it your business.
If a vendor proposes fine-tuning as the way to make a model know your documents, that is worth pushing back on. It usually means a rebuild each time your content changes, no citations, and no way to keep one customer's data away from another's.
Answered before you ask
Should I fine-tune a model on my company documents?
Almost certainly not, and this is the most common expensive misunderstanding in applied AI. Fine-tuning adjusts how a model behaves — its tone, its format, its approach to a narrow task — rather than reliably installing facts you can query later. Train on your documents and you get a model that sounds like your documents while still inventing specifics, with no way to cite a source, no way to update a fact without retraining, and no way to prevent one user from receiving information intended for another. Retrieval solves all four: the documents stay in a store you control, the relevant passages are fetched at the moment of the question, answers can point at where they came from, and correcting a wrong answer means editing a file. Use retrieval for what the model should know, and reserve fine-tuning for how it should behave.
How much data do I need for each approach?
They have very different thresholds. Retrieval works from whatever you have — a single handbook is enough to be useful, and quality scales with how well written and how current the material is rather than with volume. There is no minimum dataset and no training step, so you can start narrow and add sources as you go. Fine-tuning needs consistent examples of the input and the exact output you want, and the practical floor is in the hundreds rather than the dozens, with quality and consistency mattering far more than quantity: a few hundred carefully curated pairs beat several thousand inconsistent ones, because contradictions in the training set teach contradiction. This is another reason to ship retrieval first — a live system produces real examples of correct output, which is exactly the dataset a later fine-tune would need.
Does my data get sent to the AI provider or used for training?
With retrieval, your documents stay in a store you control and only the passages relevant to a given question are included in that request. Whether those passages leave your infrastructure depends on where the model runs: a hosted API means the request goes to the provider, while a self-hosted open-weights model means nothing leaves at all. The major providers offer business terms under which API data is not used to train their models and is retained only briefly, and that should be checked and written down rather than assumed. Where the requirement is strict, we build against a self-hosted model — it costs more and is usually a step down in capability, and for regulated work that is often the right trade. Fine-tuning is a different question again, because your examples are used to alter model weights, so read the terms on the specific fine-tuning service carefully before sending anything sensitive.
How do I know whether the system is actually any good?
By measuring it against a set of questions with known answers, written before launch and kept as the thing you test every change against. Vibes do not survive contact with a system that is right eighty per cent of the time, because the wrong twenty per cent is fluent and confident. We measure two things separately: whether the correct source material was retrieved at all, and whether the answer generated from it was right. Separating them tells you where to spend — a low retrieval rate means the problem is chunking, search or content, and no amount of model upgrading will fix it. Alongside that we track how often the system correctly says it does not know, because a system that never declines is a system that is guessing, and for most business uses a clean "I don't have that" is worth more than a plausible invention.