Every time you send a message, the model receives the whole conversation as one block of tokens. That block has a maximum size, called the context window. Everything counts against it: the system prompt, the chat history, any documents you paste in, and the model's own output. Once the total passes the limit, something has to give.
What fills the window#
The window is measured in tokens, not words or characters. Input and output share it, so a long answer leaves less room for everything else.
The model is stateless: its weights are frozen after training, so it remembers nothing between calls. The app resends the conversation each time, and that resent text is what fills the window.
What happens at the limit#
The API rejects a request outright if the input alone exceeds the window. Most of the time the limit arrives slowly instead. A conversation grows turn by turn, and the app has to choose what to drop.
There are three common strategies. Each trades memory, cost and fidelity against the others, and none of them is free.
| Strategy | What it keeps | What you lose |
|---|---|---|
| Sliding window | The last N turns | Everything before the cut |
| Summarization | A short summary of old turns | Specific details |
| Retrieval | Only the old turns that match the question | Anything the search fails to find |
Sliding window#
Keep only the last N turns. This is the simplest option. The cost is that the model forgets everything before the cut, so if you ask about something from the start of a long chat, it has no access to it.
Summarization#
Compress older turns into a short summary and keep that in their place. The model keeps a rough picture of what happened. Specific details, such as an exact number or a name, are lost.
Retrieval#
Store old turns outside the context and pull back only the ones that matter for the current question. If your question connects to something said three hours ago, that part comes back in. If it does not connect, it stays out, so it uses no context space and adds no cost.
The trade-off is that retrieval is not perfect. If the relevant old turn is not retrieved, the model acts as if it never happened. This is RAG applied to chat history.
Cost and the hidden quality problem#
Cost is easy to see. You pay for every input token on every request, not only the new ones. A conversation that is 10,000 tokens long costs 10,000 input tokens each time you send a message, so long chats get expensive fast.
The quality problem is less obvious. You might assume a bigger window means the model uses all of it equally well. It does not.
Research found that models use information at the start and end of the context much better than information buried in the middle. This is called the "lost in the middle" problem. If you paste a long document and the answer sits in the middle, the model may miss it or use it weakly, even though the text is there.
A 200,000-token window therefore does not mean the model reliably uses 200,000 tokens of information. It means that much text fits. How well the model uses the middle is a separate question, and the honest answer is: less well.
Window size tells you what fits. It does not tell you what the model will use well.
This is one more reason retrieval helps. Instead of dumping everything in and hoping, you place only the most relevant chunks where the model uses them best.
Long context or RAG#
The same trade-off decides how to give a model your documents. The two approaches fit different situations.
| Situation | Choose |
|---|---|
| Small, stable set of documents, such as a 10-page spec that never changes | Put it straight in the prompt (long context) |
| Large, frequently changing, or more than fits in the window | Use RAG |
With a small, stable set, the prompt approach is simple. There is no retrieval to build, and the whole thing fits comfortably.
With a large set, putting everything in is not practical. A 10,000-page knowledge base cannot go into every request. Even if it fit, the lost-in-the-middle problem would hurt quality. The rule of thumb: small and stable, put it in the prompt. Large or changing, use RAG.
Short line#
The context window is the model's only working memory for a single request. Everything counts against it, it resets between calls, and quality drops for information buried in the middle of a long context.
Key takeaways#
- The context window is the maximum number of tokens the model sees in one request. It covers the system prompt, history, pasted documents and the output.
- When a chat outgrows the window, the app can slide, summarize or retrieve. Each one trades memory, cost and fidelity.
- You pay for every input token on every request, so long conversations cost more each turn.
- Models use the start and end of the context better than the middle ("lost in the middle").
- Small and stable documents can go in the prompt. Large or changing ones need RAG.