RAG vs fine-tuning: which one your project needs

Teams often ask whether to use retrieval-augmented generation (RAG) or fine-tuning, as though one had to win. The two address different gaps. RAG changes what the model can see when it answers. Fine-tuning changes how the model behaves. Start by working out which of the two you are missing.

What each one changes

With RAG, your system searches your own documents when a question arrives, puts the best passages into the prompt, and asks the model to answer from them. In most business setups the base model is not retrained. The name comes from a 2020 paper by Lewis and colleagues, which paired a pretrained language model with a dense vector index of Wikipedia that a neural retriever searched at answer time (Lewis et al., 2020).

Fine-tuning trains the model on examples so its weights shift. The model then behaves differently on every request, with no retrieval step. Parameter-efficient methods shrink what has to be trained. LoRA freezes the pretrained weights and trains small added matrices instead (Hu et al., 2021). QLoRA applies that idea to a quantized model, so that a 65-billion-parameter model can be tuned on a single 48 GB GPU (Dettmers et al., 2023).

If the problem is missing knowledge, use RAG

A model does not know your pricing sheet, last week's policy change, or a contract signed yesterday. Fine-tuning looks like an obvious fix, and one careful comparison points the other way. Ovadia and colleagues compared the two on knowledge-intensive tasks. Unsupervised fine-tuning gave some improvement, but RAG consistently did better, both for facts the model had seen in training and for entirely new ones. They also found that models struggle to learn new facts through unsupervised fine-tuning (Ovadia et al., 2023).

Beyond the benchmark, three practical points favor retrieval:

• Documents change. With RAG you update the index. With fine-tuning you retrain.
• Answers need sources. A RAG system can show which passage an answer came from. A tuned model has no built-in way to point to a source.
• Access rules differ by person. A retrieval layer can filter documents by who is asking. Anything a model learned in training can surface for any user who queries it.

If the problem is behavior, consider fine-tuning

Fine-tuning earns its cost when the gap is in how the model responds and no prompt closes it. Typical cases:

• A fixed output format that the base model gets wrong often enough to matter.
• A house style or tone that is hard to describe but easy to show with examples.
• A narrow task, such as sorting support tickets into your own categories, where a smaller model tuned for that one job may be enough, and cheaper to run per request.

Fine-tuning needs clean, correctly labeled examples that cover the cases you care about, and a test set that stays out of training. Without a baseline score from the untuned model, you cannot say whether training helped.

A quick way to decide

Ask what a wrong answer looks like.

1. The answer is wrong because the fact is missing or out of date. Use RAG.
2. The answer is factually fine but in the wrong format, tone, or shape. Try a better prompt and examples first. If that fails on a measured test, consider fine-tuning.
3. The answer is wrong because the model reasons badly about your domain. Neither technique is aimed at this problem. Test a stronger base model first.

Prompting comes before both. A clear prompt with a few worked examples is cheaper than either technique, so try it first.

Using both

The two are not exclusive. A common setup retrieves current documents with RAG and uses a tuned model that follows your output format. Add the second layer only after the first is measured. Each added layer makes failures harder to trace.

Long context does not remove the choice

Some teams skip retrieval and paste whole documents into a long prompt. That works for small collections. It gets weaker as inputs grow. One study found that model performance is often highest when the relevant information sits at the start or end of the input, and drops when it sits in the middle (Liu et al., 2023). Retrieval shortens the input by choosing a few relevant passages, but where those passages sit still matters, so order them deliberately and test the order against your own questions.

A worked example

This is an illustration, not a client result. Imagine a support assistant for a software company. You run twenty real questions through the base model with a good prompt and read the failures. Some answers quote a refund policy that changed last quarter. Some describe a feature that no longer exists. Those are missing or outdated knowledge, so they point to RAG. A few answers are correct but ignore the required reply format, which points to examples in the prompt first and to fine-tuning only if that fails. One answer gets a plan comparison wrong even with the right documents in front of it, which points to testing a stronger model.

The value is in the sorting. Three groups of failures call for three different fixes, and running the same twenty questions again after each fix shows which one worked.

What to do next

Write down twenty real questions your users ask and the answers you would accept. Run them through the base model with a good prompt. Sort the failures into the three groups above. The largest group tells you what to try first.

If most failures are missing knowledge, our RAG development services page describes how we build and measure retrieval, and how to evaluate a RAG system covers the measuring. If the failures are about behavior and survived prompt work, see LLM fine-tuning services, which also says when we would talk you out of it.