Getting the right chunks into the prompt is only half the job. You also have to tell the model how to use them. Left alone, the model blends the retrieved content with its training knowledge, and that brings hallucination right back.
Three rules#
Put three rules in the system prompt. Each closes a different gap.
Answer only from the retrieved context#
The model must not add things it knows from training. Training knowledge may be out of date, wrong for your domain or simply invented, and once it is blended in, no one can tell which parts came from your documents.
Cite sources#
Give each chunk an ID in the prompt, such as [doc_1], and tell the model to cite the ID for every claim. This makes answers verifiable, and it forces the model to ground each statement in something concrete.
Say "I don't know" when the context does not cover it#
Without this rule, the model fills gaps with guesses. With it, a missing answer becomes an honest signal: retrieval did not find the right content. That signal is useful for debugging, as the common failures note shows.
Putting it in the prompt#
The chunks go in with their IDs, wrapped in delimiters so the model can tell sources from instructions.
Answer the question using only the documents below.
Cite the document ID, like [doc_1], for every claim.
If the documents do not contain the answer, say "I don't know."
<documents>
<doc id="doc_1">Refunds are accepted within 30 days of purchase.</doc>
<doc id="doc_2">Gift cards are not refundable.</doc>
</documents>| Rule | Gap it closes |
|---|---|
| Answer only from context | Training knowledge mixed into the answer |
| Cite sources | Claims no one can check |
| Say "I don't know" | Guesses when retrieval missed |
Together, these rules turn the model from a generator that happens to have some context into a grounded reader that only speaks from what it was given.
Short line#
Tell the model to answer only from the retrieved chunks, cite a chunk ID for every claim, and say "I don't know" when the chunks do not cover the question. That turns it into a grounded reader instead of a generator.
Key takeaways#
- Without explicit rules, the model blends retrieved content with training knowledge.
- Rule 1: answer only from the retrieved context.
- Rule 2: give each chunk an ID and require a citation for every claim.
- Rule 3: say "I don't know" when the context does not cover the question.
- "I don't know" is an honest signal that retrieval missed, not a failure to hide.