The problem both approaches solve
Off-the-shelf AI models know a lot about the world but nothing about your business: your product catalogue, internal policies, pricing or the way your support team writes. Out of the box, they’ll guess — and confident guesses about refund policies are how companies end up in the news. RAG and fine-tuning are the two main ways to close that gap. (New to how models work? Start with How Do LLMs Work?.)
What is RAG?
Retrieval-augmented generation was named in a 2020 research paper from Facebook AI Research (now Meta AI). The idea is simple: before the model answers, a search step finds the most relevant passages from your own documents and adds them to the prompt. The model then writes its answer using that material.
- Your documents are split into short passages and indexed — usually as embeddings in a vector database, sometimes alongside ordinary keyword search.
- A user asks a question.
- The system retrieves the handful of passages most related to the question.
- Those passages and the question go to the model, with an instruction to answer from them.
- The answer comes back, ideally with links to the passages it used.
Think of it as an open-book exam. The model doesn’t need to memorise your handbook; it just needs to read the right page at the right time.
What is fine-tuning?
Fine-tuning continues training an existing model on a set of your own examples — typically hundreds to thousands of input-and-ideal-output pairs. The model’s internal parameters change, so the new behaviour is built in and doesn’t need to be spelled out in every prompt. Modern techniques such as LoRA (low-rank adaptation) adjust only a small fraction of the parameters, which makes fine-tuning far cheaper than it used to be.
This is the studying approach: practise on enough examples and the patterns become second nature. It’s excellent for teaching *how* to respond, and unreliable for teaching *what is true*.
RAG vs. fine-tuning: side by side
| RAG | Fine-tuning | |
|---|---|---|
| Best for | Facts, documents, up-to-date knowledge | Tone, format, specialised tasks |
| Updating knowledge | Edit or add a document — live immediately | Retrain the model |
| Shows sources | Yes, can cite the passages used | No |
| Risk of made-up facts | Lower, if retrieval finds the right text | Unchanged or higher for new facts |
| Upfront effort | Organise and index documents | Collect and clean training examples |
| Ongoing cost | Longer prompts on every request | Training runs; sometimes higher hosting cost |
| Access control | Can filter documents per user | Everything trained in is available to everyone |
When RAG is the right choice
- Customer support assistants that must quote current policies and product details.
- Internal knowledge search across wikis, contracts, manuals or past tickets.
- Anything that changes weekly — prices, stock, schedules, regulations.
- Regulated settings where every answer must be traceable to a source document.
- Mixed permissions, where different staff may see different documents.
When fine-tuning is worth it
- A consistent brand voice that long style instructions don’t reliably produce.
- Strict output formats, such as specific JSON structures or report templates.
- Narrow, repetitive tasks like classifying tickets or extracting fields from invoices, where a smaller fine-tuned model can be faster and cheaper than a large general one.
- Specialised language — legal, medical or technical phrasing the base model handles poorly.
Can you use both?
Yes, and mature systems often do. A model fine-tuned to write in your support team’s style and format, fed with up-to-date passages through RAG, gets the best of both. The sensible order is to add fine-tuning on top of a working RAG system, not the other way round.
A simple decision checklist
- Have you tried better prompts? Clear instructions and a few examples in the prompt solve more problems than people expect, at almost no cost.
- Is the problem missing or changing information? Use RAG.
- Is the problem style, format or a narrow repeated task? Consider fine-tuning.
- Do you need to show where answers came from? RAG.
- Do you have a few hundred high-quality examples of ideal outputs? If not, you’re not ready to fine-tune.
What it means for a small business
For most small and mid-sized companies, the practical answer is RAG — often without building anything. Many AI assistants and help-desk tools now let you upload documents or connect a knowledge base and handle retrieval for you. Keep those documents accurate and current: a RAG system can only be as good as the files it searches. If you’re working out costs, our guide to tokens and context windows explains how longer prompts affect your bill.
