When a user sends a message, the wait for the answer has two phases. Each phase grows with a different thing, so you need to know which one you are paying for before you try to make a feature faster.
Prefill and decode#
| Phase | What the model does | What it sets | Grows with |
|---|---|---|---|
| Prefill | Reads your whole prompt in one parallel pass | Time to first token | Prompt length |
| Decode | Writes the answer one token at a time | The rest of the wait | Output length |
Prefill is fast because the model processes all input tokens together. Decode is slow because each output token needs its own full pass through the model (see Next-token prediction).
Output length dominates#
In practice, output length decides most of the total time. A 1,000-token answer takes roughly 5 to 10 times longer than a 100-token answer. Shortening or capping the output is usually the biggest latency win you have.
Four levers#
| Lever | What to do | Why it helps |
|---|---|---|
| Stream the output | Send tokens to the user as they are generated. | The model does the same work, but the user reads text from the first token instead of waiting for the end. |
| Cap output length | Set max_tokens on every request. |
A vague prompt cannot produce 2,000 tokens when 200 would do. |
| Use a smaller model | Route easy tasks to a small, fast model. | A small model can be 5 to 10 times faster than a large one. |
| Cache repeated prefixes | Use provider prompt caching for a prefix that repeats on every request. | The provider skips most of the prefill work for the cached part. |
Streaming is the easiest win, so make it the default for any user-facing feature. The system prompt is the main candidate for caching, because it is the same on every request.
Short line#
Latency is prefill (reading the prompt) plus decode (writing the answer one token at a time), and decode usually dominates. Stream the output, cap max_tokens, route easy tasks to a smaller model, and cache the repeated prompt prefix.
Key takeaways#
- Prefill reads the prompt in one parallel pass and sets time to first token. It grows with prompt length.
- Decode writes one token at a time and grows with output length.
- Output length dominates: a 1,000-token answer takes roughly 5 to 10 times longer than a 100-token one.
- Streaming does not make the model faster, but the user sees text immediately. Make it the default.
- Set
max_tokenson every request. - A small model can be 5 to 10 times faster; use it where it is good enough.
- Prompt caching cuts prefill work for a repeated prefix such as the system prompt.