When an LLM feature does not work well enough, there are several things you can change: the prompt, the knowledge the model gets, the model itself, or whether you use a model at all. Each fixes a different kind of problem. Picking the wrong one is the most expensive mistake in AI engineering, so start by naming the problem.
Name the gap first#
| Gap | What it looks like | Fix |
|---|---|---|
| Knowledge | The model lacks the right facts: too old, private, or never seen | RAG or tools |
| Behavior | The model knows enough but answers in the wrong format, tone or style | Better prompt, then fine-tuning |
| Capability | The task is too hard for the model | A stronger model, or split the task into smaller steps |
| None | Rules or plain code already solve it | Do not use AI |
Fine-tuning to fix a knowledge gap is the single most common mistake. The rest of this page explains why.
The order to try#
Try fixes in this order: a better prompt, then RAG, then fine-tuning. Each step costs much more and takes much longer to iterate on than the one before it, so jumping ahead wastes time and money.
- Better prompt. Clear instructions, delimiters, examples and step-by-step reasoning cost almost nothing and fix most problems. If you have not tried them properly, you are not ready to move on.
- RAG. If the model is wrong because it lacks knowledge, give it the knowledge at question time. This is cheaper and faster to change than training.
- Fine-tuning. Only after prompting and RAG have really been tried and cannot solve the problem.
RAG for knowledge#
RAG is the right tool when the model does not have the right facts.
| Need | Why RAG fits |
|---|---|
| Fresh data | The model has a knowledge cutoff. RAG retrieves current information at question time. |
| Private data | The model never saw your internal documents, product specs or customer data. RAG brings them in without retraining. |
| Citations | RAG puts specific chunks in the prompt, so the model can cite where an answer came from and the user can check it. |
What RAG does not do is change how the model behaves. If the model keeps answering in the wrong format or tone, that is not a knowledge problem, and more retrieved documents will not fix it.
Fine-tuning for behavior#
Fine-tuning trains the model further on your own examples. It is the right tool when the problem is how the model behaves, not what it knows.
| Use | How it helps |
|---|---|
| Format and style | When prompting cannot reliably produce a very specific structure or a consistent tone, hundreds of examples of exactly that output make it more reliable. |
| Narrow, repetitive tasks | A model that does one task thousands of times a day can learn it, so it needs shorter prompts and fewer tokens. |
| Distillation | A large, expensive model produces high-quality outputs for your task. A small, cheap model is fine-tuned on them and keeps much of the quality at a fraction of the cost. |
Why fine-tuning does not reliably add facts#
Each fact appears in only a few training examples, so the model learns it unreliably. Fine-tuning on new facts can even increase hallucination. It has other limits too:
| Need | RAG | Fine-tuning |
|---|---|---|
| Update one fact | Edit the document | Retrain |
| Cite a source | ✓ | ✗ |
| Per-user permissions | ✓ (filter at retrieval, see Permissions) | ✗ (everything is baked into the weights) |
If the problem is knowledge, use RAG. Fine-tuning is for behavior.
When not to use AI#
Sometimes the right answer is no model at all.
| Situation | Why plain code wins |
|---|---|
| Rules and logic already work | If the task fits SQL, a regex, a lookup table or a decision tree, that is faster, cheaper, deterministic and auditable. |
| Exact math | LLMs are bad at arithmetic. Use a calculator or code. |
| Strict latency | An LLM call takes hundreds of milliseconds to seconds. A feature that needs a 10 ms response cannot use one. |
| Decisions must be auditable | In regulated industries you must explain exactly why a decision was made. A rules engine can show that; an LLM cannot. |
| Costly mistakes with no review | If a wrong answer causes real harm and no human checks it, the LLM's error rate is too high. |
The test is one question: would a simpler, deterministic solution work here? If yes, use it.
Hybrid systems#
Real production systems layer these approaches instead of picking one:
- Rules first. Deterministic code handles the easy, well-defined cases. It is fast, cheap and auditable.
- LLM for fuzzy cases. When the input is ambiguous or open-ended and no rule covers it, the model handles this long tail.
- Human review when confidence is low. When the model is not sure, the case goes to a person instead of showing a possibly wrong answer.
For example, a document classification system uses a regex to catch documents that match a known pattern exactly. An LLM classifier handles the ambiguous ones. Anything the LLM scores below a confidence threshold goes to a human reviewer.
A hybrid is more work to build than a pure LLM solution. You maintain rules, tune thresholds and run a review queue. But a pure LLM solution is fragile and expensive for cases that rules handle cleanly. The choice comes down to volume and stakes.
Short line#
Name the gap first. Try a better prompt, then RAG for knowledge gaps, then fine-tuning for behavior gaps, and never fine-tune to add facts. Use plain code when rules work, the math is exact, latency is strict or decisions must be auditable. In production, rules handle the easy cases, the LLM handles the fuzzy ones, and humans review low-confidence outputs.
Key takeaways#
- Name the gap: knowledge → RAG or tools, behavior → prompt then fine-tuning, capability → stronger model or smaller steps, none → no AI.
- Order: better prompt, then RAG, then fine-tuning. Each step is more expensive and slower to iterate.
- RAG fits fresh data, private data and citations. It does not change behavior.
- Fine-tuning fits format and style, narrow repetitive tasks and distillation.
- Fine-tuning does not reliably add facts and can increase hallucination. It cannot cite, update one fact or apply per-user permissions.
- Use plain code for rules, exact math, strict latency, auditable decisions, and costly unreviewed mistakes.
- Hybrid: rules first, LLM for the long tail, humans for low confidence.