Most companies that want “ChatGPT for our business” are really asking one of two things. Either they want an AI that knows their information (policies, products, contracts, tickets), or they want an AI that behaves in a particular way (writes in their format, classifies their documents, follows their rules). The first is a job for retrieval-augmented generation (RAG). The second is where fine-tuning can help.
Choosing the wrong one is one of the most common reasons AI pilots disappoint. Here is how to tell them apart.
What RAG does
Retrieval-augmented generation keeps your knowledge outside the model. When someone asks a question, the system first searches your content for the most relevant passages, then gives those passages to the language model with the question, and the model writes an answer from them.
A typical RAG system has four parts:
- Ingestion: documents are collected from where they live (shared drives, wikis, CRMs, ticketing tools), cleaned and split into passages.
- Indexing: each passage is stored in a search index, usually combining keyword search with vector search (embeddings) so both exact terms and meaning are matched.
- Retrieval: at question time, the most relevant passages are found, filtered by what that user is allowed to see.
- Generation: the model answers using only those passages and cites them.
Because the knowledge sits in an index, updating it is as simple as re-indexing a changed document. Answers can show their sources, so people can check them. And access rules can be enforced before the model ever sees a passage.
What fine-tuning does
Fine-tuning changes the model itself. You train an existing model further on hundreds or thousands of examples of the inputs and outputs you want, and its weights shift so it produces that kind of output more reliably.
Fine-tuning is good at:
- Consistent format and style: always returning a specific JSON structure, a report layout, or your brand’s tone.
- Narrow, repeated tasks: classifying tickets into your categories, extracting fields from your document types, routing requests.
- Efficiency: a smaller fine-tuned model can sometimes match a larger general model on one task, at lower cost and latency.
It is weak at knowledge. Facts absorbed during fine-tuning are hard to update (you retrain), impossible to cite, and can be blended or recalled wrongly. If your price list changes next month, a fine-tuned model will not know.
Side by side
| Question | RAG | Fine-tuning |
|---|---|---|
| Main purpose | Give the model your knowledge | Change how the model behaves |
| Information changes often | Handles it: re-index the document | Needs retraining |
| Answers show sources | Yes | No |
| Respects user permissions | Yes, at retrieval time | No, knowledge is baked in |
| Needs training data | No, needs your documents | Yes, many good examples |
| Typical first step | Yes | Rarely |
How to decide
Ask what is going wrong today.
“The AI doesn’t know our stuff.” Use RAG. This covers internal knowledge assistants, customer support answers grounded in your help centre, policy and contract Q&A, and search across scattered documents.
“The AI knows enough but doesn’t do the task the way we need.” First try better instructions and a few examples in the prompt. If that still fails consistently, fine-tune for that specific behaviour.
“We need both.” Common in practice: RAG supplies the facts, and a fine-tuned or carefully prompted model turns them into the exact output format a downstream system expects.
Where RAG projects go wrong
RAG is the right default for most business knowledge, but its quality depends on unglamorous details:
- Messy source content. Outdated, duplicated or contradictory documents produce confident wrong answers. Clean-up and ownership of content matter as much as the model.
- Poor chunking. Splitting documents in the wrong places separates a rule from its exceptions. Respect headings, tables and sections.
- Retrieval that misses. If the right passage is not retrieved, no model can answer well. Hybrid search, re-ranking and good metadata fix most of this.
- No permissions. An assistant that can quote any document to anyone is a data leak. Filter by access rights before generation.
- No evaluation. Build a test set of real questions with known good answers and run it on every change.
Evaluate before you scale
Whichever approach you choose, decide how you will measure success before building. A useful evaluation set has real questions from real users, the correct answer or source for each, and the edge cases you are worried about. Track answer accuracy, whether the right sources were used, refusals when the answer is not available, and cost per answer. Then pilot with a small group and compare against how the work is done today.
That discipline, more than the choice of technique, is what moves an AI pilot into production.