When a RAG answer is wrong, the cause usually falls into one of four buckets. Each bucket has a different fix, so the most important skill is working out which one you are in. Fixing the prompt when the problem is retrieval only wastes time.
The right document is not retrieved#
The query embeds in a part of the vector space that does not land near the right chunk. Or the chunk was never indexed, or the index is stale. The model never had a chance to answer correctly.
This is a retrieval failure. Hybrid search, better chunking and query rewriting all help. Query rewriting means turning the user's question into a clearer search query before you embed it.
A chunk cuts the answer in half#
The relevant information sits right on a chunk boundary and gets split across two chunks. Neither chunk alone is enough to answer.
Overlap reduces this, and small-to-big retrieval mostly eliminates it, because the model receives the whole parent section.
The model ignores the context#
The right chunks are in the prompt, but the model answers from its training knowledge anyway, or blends the two. This is a generation failure, not a retrieval failure.
Stronger answer rules are the fix, along with a model that follows instructions well.
The index is out of date#
A document was updated or deleted, but the index was not. The system retrieves the old version, or content that should no longer exist.
The fixes are operational: re-index a document incrementally when it changes, and delete its chunks when the source is deleted.
Diagnosing which bucket you are in#
| Failure | Phase | Fix |
|---|---|---|
| Right document not retrieved | Retrieval | Hybrid search, better chunking, query rewriting |
| Chunk cuts the answer in half | Indexing | Overlap, small-to-big retrieval |
| Model ignores the context | Generation | Stronger answer rules, better instruction following |
| Index out of date | Operations | Incremental re-index, delete chunks with their source |
When an answer is wrong, first check whether the right chunk was retrieved at all. Log the retrieved chunks for every question so you can check.
Short line#
Most RAG failures are one of four: the right chunk was not retrieved, a chunk split the answer, the model ignored the context, or the index was stale. Diagnose first, starting with whether the right chunk was retrieved.
Key takeaways#
- Retrieval failure: the right chunk never reached the prompt. Use hybrid search, better chunking, query rewriting.
- Split answer: the answer sits on a chunk boundary. Use overlap or small-to-big retrieval.
- Generation failure: the model ignores the chunks. Strengthen the answer rules.
- Stale index: re-index on change and delete chunks with their source.
- Check retrieval first. Fixing the prompt does nothing for a retrieval problem.