Temperature is a setting that controls how random a model's output is. It does not change what the model knows. It changes how boldly the model picks among the tokens it already considers likely. Knowing what it does helps you choose a good value for each task, and helps you avoid expecting it to fix problems it can't.
From scores to probabilities#
When the model finishes a pass through its network, it has a score for every token in its vocabulary. These scores are called logits, and a higher score means the model thinks that token is more likely to come next. (See Tokens for what a token is.) Before the model picks one, the scores are converted into probabilities.
Temperature is the number that controls this conversion. At temperature 1, the probabilities reflect the raw scores fairly directly. Lower values sharpen them, and higher values flatten them. The pick itself is the step described in Next-token prediction.
Sharpened vs flattened#
Imagine the model is choosing the next word after "The capital of France is". The scores might give Paris 95%, Lyon 3%, London 1%, and the rest spread across thousands of other tokens. The chart below shows how temperature changes that spread.
Low temperature, say 0.1, sharpens the distribution. Paris goes to about 99.9% and everything else collapses. The model almost always picks the top token, so the output is focused, consistent and predictable. If you run the same prompt twice, you get nearly the same answer.
High temperature, say 1.5, flattens it. Paris might drop to 60%, Lyon and London rise, and tokens that were nearly zero now have a small but real chance. The model picks from a much wider spread, so the output is more varied and more surprising. It is also more likely to go off the rails.
Temperature 0 is the extreme case: always pick the highest-scoring token, with no randomness at all.
The percentages here are examples to show the shape of the change. Real values depend on the model and the prompt.
Choosing a value#
Each end of the range suits different work. The table sums it up.
| Low temperature | High temperature | |
|---|---|---|
| Output | Focused, consistent, predictable | Varied, creative, surprising |
| Use it for | Extraction, classification, structured output | Brainstorming, several different options, creative writing |
| Risk if pushed too far | Repetitive and bland, always the safe obvious token | Nonsense or drifting off topic |
Use low values when you want the same reliable answer every time. Use high values when you want range. "Generic" describes low temperature gone wrong, not high temperature, which is varied rather than generic.
The temperature 0 gotcha#
You might assume temperature 0 means perfectly deterministic output: the same prompt gives the same answer every time. In practice it is not quite true.
Providers run many requests in parallel and batch them together for efficiency. Floating-point arithmetic on GPUs is not perfectly reproducible across different batch sizes and orderings. The tiny numerical differences add up through billions of operations, and now and then a different token wins by a hair.
So temperature 0 means nearly deterministic, not perfectly deterministic. Outputs are very stable in practice, but don't build a system that breaks when two runs differ by a word.
What temperature does not do#
Temperature doesn't add knowledge or improve reasoning. It only reshapes the probability distribution over tokens that already exist. A model that is wrong at temperature 1 is still wrong at temperature 0, just more consistently wrong.
A common trap is the question "Would lowering the temperature reduce hallucinations?" The honest answer is only slightly, and indirectly. Low temperature keeps the model on its highest-confidence tokens, so it is less likely to wander into unlikely territory. But if the most likely token is already wrong, low temperature locks that in. A model's job is to produce plausible text, as covered in Next-token prediction, and plausible is not always true. The real fixes for hallucinations are grounding (giving the model the facts) and verification, not temperature.
Short line#
Temperature controls how peaked or flat the token probability distribution is: low for focused and consistent, high for varied and creative. It reshapes probabilities and does not add knowledge.
Key takeaways#
- The model scores every token (logits), and temperature controls how those scores become probabilities.
- Low temperature sharpens the distribution. High temperature flattens it.
- Use low for extraction, classification and structured output. Use high for brainstorming and creative work.
- Too low gets bland and repetitive. Too high drifts into nonsense.
- Temperature 0 is nearly deterministic, not perfectly, because of GPU floating-point and batching effects.
- Temperature adds no knowledge. Grounding and verification, not low temperature, are the real fixes for hallucinations.