Context windows have grown very large, so it is fair to ask why you would build RAG at all. Sometimes you should not. The choice comes down to how much content you have, how often it changes, and who is allowed to see it.
When long context is enough#
For a small, stable set of documents, put them in the system prompt. It is simpler, there is no retrieval pipeline to build, and the model has everything it needs on every request. This is the rule of thumb from the context window note: small and stable goes in the prompt.
When RAG wins#
RAG is the better choice when any of these is true:
- Too large: the documents do not fit in the context window.
- Changing often: a hardcoded prompt would go stale.
- Citations needed: answers must point to specific sources.
- Cost matters: sending 100,000 tokens of documents on every request is expensive. See cost.
Lost in the middle also applies. A model with a 200,000-token window does not use all of it equally well, and the middle gets the least attention. Retrieval sends only the few chunks that matter, so the right content is easy for the model to use.
The trade-off#
The trade-off is complexity against scale and freshness. RAG adds a lot of parts to build and run:
- An embedding model
- A vector database
- A chunking pipeline and an indexing job
- A retrieval step
- The operational work to keep the index fresh
| Situation | Choice |
|---|---|
| Small internal tool, a few stable documents | Long context |
| Thousands of documents | RAG |
| Users with different access levels | RAG, with permission filters |
| Content that changes often | RAG |
For a production system with thousands of changing documents and users with different access levels, RAG is the only practical option.
RAG reduces hallucination, it does not remove it#
The model can still go beyond what the retrieved chunks say, ignore the context, or invent answers for parts that retrieval did not cover. The answer rules still apply on top of RAG, and so do the checks in catching hallucinations.
Short line#
Put a small, stable set of documents straight in the prompt. Use RAG when documents are large, change often, need citations, need per-user access or cost too much to send every time. RAG reduces hallucination but does not remove it.
Key takeaways#
- Small and stable documents: put them in the system prompt and skip RAG.
- RAG wins on size, freshness, citations, access control and cost.
- Long context suffers from lost in the middle. Retrieval sends only what matters.
- RAG adds an embedding model, a vector database, chunking, indexing, retrieval and index upkeep.
- RAG reduces hallucination. Answer rules and output checks still apply.