Next-Token Prediction

At its core, an LLM does one thing: it looks at all the tokens so far and predicts which token should come next. Every answer you get, from a one-word reply to a long essay, is built from that single step repeated. Knowing this explains both why LLMs are so flexible and why they sometimes state wrong things with full confidence.

How generation works#

Generation is a loop. The model reads the text so far, scores every possible next token, picks one, and runs again.

1. You send the prompt, which arrives as token IDs (see Tokens).
2. The model produces a probability for every token in its vocabulary, which means tens of thousands of candidates. For "The cat sat on the", it might give "mat" a high probability, "floor" a bit less, and "banana" almost nothing.
3. One token is picked from those probabilities and added to the end. How the pick is made is covered in Temperature.
4. The whole thing runs again with that new token included.

The model keeps looping until it produces a special "stop" token or hits the output length limit. A full answer is just this loop running hundreds of times.

Why models make things up#

Look at what the loop is optimizing for: the most plausible next token, not the true one. During training, the model learned which tokens tend to follow which, from patterns across a huge amount of text (see Training).

So when you ask a question, the model produces what a likely answer looks like. There is no separate step where it checks a fact against a source. The loop is prediction, start to finish.

Most of the time plausible and true line up, because true things appear often in the training text. They come apart when the real answer was rare or missing in the data. The model still produces something, and a fluent, confident, wrong answer can be the most plausible-looking continuation.

A classic example is asking for a source on some topic. You may get a realistic author name, a believable paper title and a year, none of which exist. That is exactly what citations look like, so those tokens score high.

The loop only moves forward#

Once a token is picked and appended, it is permanent. The model cannot go back and edit something it already wrote.

Every new token is predicted from everything before it, including the model's own earlier output. So if the model writes a wrong claim early, the following tokens are conditioned on that claim. The most plausible continuation of a wrong statement is usually more text that supports it, so the model tends to build on the error and even justify it. This is called snowballing.

That is why an early mistake in a long answer is costly: the rest of the answer inherits it. It is also why asking a model to reason before it answers helps. The reasoning tokens come first, so the final answer is conditioned on worked-out steps instead of a snap guess. Reasoning models are built to do this automatically.

The trade-off#

The same loop that makes models unreliable also makes them useful. Because the model continues text, almost any task can be phrased as a continuation: answering a question, writing code, summarizing, translating or extracting data. One model handles all of them without task-specific programming, and that flexibility comes directly from next-token prediction.

The cost is that fluent does not mean correct. The model has no built-in way to check itself, and it sounds just as confident when it is wrong as when it is right.

So in engineering, verification gets added around the model instead of being expected from it. These are the usual layers.

• Grounding: retrieved documents (RAG) give the model the real facts to continue from.
• Tools: a calculator or a database lookup handles exact answers.
• Checks: validation in code catches bad output.
• Human review: a person signs off when the stakes are high.

Short line#

An LLM repeatedly predicts the next token from everything so far, picks one, appends it, and repeats. It optimizes for plausible, not true.

Key takeaways#

• An LLM predicts one token at a time, scoring every token in its vocabulary at each step.
• The loop runs until the model emits a stop token or reaches the output length limit.
• The model optimizes for the most plausible next token, with no separate fact check, so fluent wrong answers happen.
• Appended tokens are permanent, and later tokens build on earlier ones, so an early error snowballs.
• Reasoning before answering helps because the answer is conditioned on worked-out steps.
• Add grounding, tools, checks and human review around the model, because it cannot verify itself.