LLM engineering
Fine-tuning, RAG or prompting: choosing without wasting a quarter
Three techniques that solve different problems and get picked for the wrong reasons. A decision order that avoids the expensive mistake.
6 min readByteWeave Studio
Teams reach for fine-tuning when the model does not know something, which is the one problem fine-tuning is worst at. They reach for RAG when the model does not behave a particular way, which retrieval will not fix either.
The two failures have a tell. If the model is missing facts, that is a knowledge problem. If it knows the facts and presents them wrongly, that is a behaviour problem. Different problems, different tools, and the expensive mistake is confusing them.
Prompting first, and further than feels respectable
A carefully structured prompt with three or four well-chosen examples gets further than most teams expect, and it is the only option you can change in a minute. Before concluding that prompting has run out, check the boring things: is the instruction ambiguous, are the examples actually representative, is the output format specified precisely enough to parse.
The reason to exhaust this first is not cost, it is iteration speed. You will learn more about the shape of your problem in two days of prompt iteration than in two weeks of assembling a training set, and that understanding is what makes the next decision correct.
RAG when the answer lives in your documents
Retrieval is the right tool when correctness depends on information the model was never trained on and that changes: your policies, your catalogue, this quarter’s pricing. It has the property fine-tuning lacks, which is that updating a document updates the system immediately, with no training run and no evaluation cycle.
It also makes answers checkable. A retrieved passage can be cited, and a user can follow the citation. In any regulated or high-consequence setting that traceability is often the requirement, quite apart from accuracy.
Fine-tuning for behaviour, format and cost
Fine-tuning teaches a model how to respond rather than what is true. It is the right tool for a consistent output structure that prompting keeps drifting away from, for a specialised tone, for a classification task with a fixed label set, and for compressing a long prompt into the weights so a smaller and cheaper model can do the job.
It is the wrong tool for facts. Teaching a model your product catalogue by fine-tuning produces a system that is confidently out of date the moment the catalogue changes, with no way to tell which parts are stale. If the information has a shelf life, retrieve it.
The order that avoids the wasted quarter
Work in this sequence: a strong prompt with examples, then retrieval if correctness depends on your own data, then fine-tuning if behaviour or unit cost is still wrong after both. Most projects stop at step two. The ones that reach step three arrive with a clear specification of what they need, which is what makes fine-tuning work when it works.
Whatever you pick, build the eval set first. Without one you cannot tell whether fine-tuning helped, and a quarter spent on training runs you could not measure is the specific failure this ordering exists to prevent.
- LLM
- RAG
- Fine-tuning
- Architecture
More reading
LLM engineering
Why your RAG demo works and your RAG product doesn’t
A retrieval demo on twenty documents proves almost nothing about the same system on twenty thousand. The four things that break in between.
6 min read
LLM engineering
How to build an LLM eval set when you have no labelled data
You cannot improve what you cannot measure, and nobody hands you a labelled set. How to build a useful one in about a day.
6 min read
Document AI
Why your invoice OCR works in testing and fails in production
The test folder is clean exports. The real intake is phone photographs. What changes between the two, and how to find out before a client does.
5 min read
Have a problem
worth solving?
Tell us what you're building. We'll help you figure out what's possible — and say so if we're not the right people for it.