The question “RAG or fine-tuning” is almost always framed wrongly. These are not two competing ways of doing the same thing. RAG answers “what does the model know”; fine-tuning answers “how does the model speak and in what format does it reply”. Confusing knowledge with behaviour causes the most expensive architecture mistakes I see.

What RAG means in practice

Retrieval-Augmented Generation is search plus generation. Your documents are split into fragments and indexed; for every question the system first finds the relevant chunks and then asks the model to answer strictly from them. The model does not memorise your policies — it reads them at answer time.

Every property of RAG follows from that: updating knowledge means uploading a new document rather than retraining; an answer can carry a link to its source; and when there is no data, the system can honestly decline. For legal, financial and medical work, that last point often matters more than elegant wording.

When fine-tuning is genuinely worth it

Fine-tuning earns its place in four situations:

  • You need a consistent voice or tone that instructions cannot pin down.
  • You need a strict output format — product listing markup, say, where any deviation breaks the import.
  • The task is narrow and repetitive and you want to cut cost noticeably: a small tuned model is cheaper than a large general one.
  • You have several thousand clean “input — correct output” examples, not a dozen.

That last point rules out most requests. Collecting and cleaning a training set costs more than it looks, and you will maintain it forever: every process change means a new dataset version and another run.

Total cost of ownership

Count a year of operation, not the build. With RAG the main costs are the vector store, re-indexing and tokens for long context. With fine-tuning they are data collection, training runs, regression testing and retraining whenever the base model changes. The second list grows faster and needs people, not just a provider account.

If your knowledge changes more often than once a quarter, a tuned model will spend its life chasing reality and always trail it.

The hybrid that usually wins

In practice the winning setup is a combination: RAG carries the facts, prompts and tools carry the behaviour, and only if a systematic formatting problem remains do you add light fine-tuning. The order matters: cheap layers first, expensive ones after.

And before fine-tuning, try the cheaper things: better document chunking, queries rewritten for search, reranking of results, and clear instructions with examples. On real knowledge bases those four steps lift quality more than training does.

How to verify the choice in a week

Collect 50 to 100 real questions with reference answers from an expert. Run them through a RAG prototype and see exactly where quality is lost: the system found no document, found the wrong one, or found the right one and answered badly. The first two are fixed by retrieval, the third by instructions or tuning. Without that breakdown, choosing an architecture is a matter of taste.