A model is trained in three stages: pretraining, instruction tuning and preference tuning. Each stage teaches something different, and together they explain why a chatbot behaves as it does. They also explain its limits: a knowledge cutoff, a habit of pleasing you, and no memory of your past chats.
Stage 1: Pretraining#
In pretraining, the model reads an enormous amount of text: web pages, books, code and articles, trillions of tokens in total. The task is the same loop from next-token prediction: predict the next token.
Here is how one step works. The model sees "The capital of France is" and guesses the next token. The real text says "Paris". If the guess was off, the model's internal numbers, called weights (billions of them), get nudged slightly so the right token becomes a bit more likely next time. This repeats across trillions of examples.
Nobody labels anything by hand. The text itself holds the answer, because the next token is right there. This is called self-supervised learning, and it is what makes training on that much data possible.
By the end, the model has absorbed grammar, facts, coding patterns and reasoning habits. All of it comes as a side effect of getting good at predicting the next token.
What pretraining produces: a base model#
The result is a base model. It knows a lot, but it does not behave like an assistant. It is a very powerful autocomplete.
Give a base model "What is the capital of France?" and it might continue with "What is the capital of Germany? What is the capital of Italy?". Text like that question often appears in quiz lists, so more questions are the most likely continuation. The model has not learned that a question is a request to answer.
It also has no sense of being helpful or careful. It continues whatever you give it, in whatever style the text suggests, including rude, wrong or harmful text, because the internet contains all of that. So the knowledge is there but the behavior is not, and the next two stages fix the behavior.
Stage 2: Instruction tuning#
The fix for a powerful autocomplete is to show the model what good assistant behavior looks like. The developer builds a dataset of example conversations, each with a prompt and an ideal response. For instance, "What is the capital of France?" is paired with "The capital of France is Paris."
People write many of these examples, and others are generated and then filtered for quality. They cover answering questions, following instructions, writing code, summarizing and declining harmful requests. The model is trained further on them with the same next-token objective, only on differently shaped data. After enough examples, it learns that when it sees a request, a helpful answer comes next. It also learns the chat format itself, with its system, user and assistant roles.
This dataset is tiny compared with pretraining: thousands to millions of examples, against trillions of tokens. That size gap matters. Instruction tuning mostly teaches behavior and format, not new knowledge, because the knowledge came from pretraining. The same idea holds later when you fine-tune a model yourself: it is good for behavior but unreliable for adding facts.
This stage is also called supervised fine-tuning, or SFT.
Stage 3: Preference tuning#
Instruction tuning teaches the model what an answer looks like. Preference tuning teaches it which answers are better. It works in three steps.
- The model generates two or more responses to the same prompt.
- People compare them and pick the better one: more helpful, more accurate, clearer or safer. Increasingly, another model does the comparing, guided by written principles.
- The model is trained to make the preferred kind of response more likely and the rejected kind less likely.
The trick is that judging is easier than writing. It is hard to write the perfect answer to thousands of prompts, but it is easy to say that A is better than B. Comparisons also capture subtle qualities that are hard to put into example answers, such as tone, honesty about uncertainty and how much detail fits.
Two methods carry names you will see often.
| Method | How it works |
|---|---|
| RLHF (reinforcement learning from human feedback) | The comparisons first train a separate reward model that scores answers. The main model is then tuned to score highly with it. |
| DPO (direct preference optimization) | A simpler method. It trains directly on the preferred and rejected pairs, with no separate reward model. |
Knowledge cutoff and frozen weights#
Training uses data collected up to a certain date, called the knowledge cutoff. The model knows nothing that happened after it. Once training finishes, the weights are frozen: the model is a fixed file of numbers that does not change when you use it.
That has three practical consequences.
| Consequence | What it means for you |
|---|---|
| It does not learn from your conversations | When a chat seems to remember you, the app resends earlier messages or stores notes and feeds them back in. The model itself is unchanged. See the context window. |
| Knowledge goes stale | Ask about something after the cutoff and the model either says it does not know or gives a plausible guess. That guess is next-token prediction filling a gap. |
| Updating knowledge is expensive | Retraining a large model costs a lot of time and money, so it happens rarely. This is the main reason RAG and tools exist: they hand the model fresh or private information at question time instead of retraining it. |
A detail worth knowing: models often do not know their own cutoff exactly, or much about themselves. Text about a model mostly gets written after its release, so little of it was in its own training data.
Side effects of training#
Preference tuning rewards answers that people or judge models like, and liked is not always the same as true. Three habits come out of that.
| Habit | Why it appears |
|---|---|
| Sycophancy | People tend to prefer answers that agree with them, so models learn to agree. Push back on a correct answer and the model may cave. Ask "Why did X fail?" when X never failed and it may explain the failure anyway. |
| Confident tone | Clear, confident answers get picked over hedged ones, so the model sounds just as sure when it is guessing. |
| Guessing over "I don't know" | If graders score a wrong answer and "I don't know" the same, a guess sometimes wins, so the model learns to guess. This feeds directly into hallucinations, covered in next-token prediction. |
Developers actively train against these habits, but they never fully go away. That is why good prompts explicitly allow "I don't know", and why you still verify outputs.
The trade-off#
Training turns raw text prediction into a helpful, safer assistant. The price is a frozen model with a knowledge cutoff that is expensive to update, plus a pull toward pleasing the user rather than being right. The engineering answers are RAG and tools for freshness, and verification for correctness.
Short line#
Pretraining on huge text teaches knowledge through next-token prediction, instruction tuning teaches it to follow requests, and preference tuning makes answers helpful and safe. The weights are then frozen at a knowledge cutoff.
Key takeaways#
- Pretraining predicts the next token over trillions of tokens, with no hand labels. It builds knowledge and gives you a base model.
- A base model is powerful autocomplete. It continues text and does not answer requests.
- Instruction tuning (SFT) teaches behavior and format with a small set of example conversations, not new knowledge.
- Preference tuning (RLHF or DPO) teaches which answers are better, because judging is easier than writing.
- Weights are frozen at the knowledge cutoff, so the model does not learn from your chats and goes stale.
- Training for approval brings sycophancy, confident tone and guessing, so verify important outputs.