A language model does not read text the way you do. Before anything reaches the model, the text is chopped into small pieces called tokens. Tokens set the price of a request, the speed of a reply and the size of the prompt you can send. They also explain several odd mistakes that models make.
What a token is#
A token is usually a piece of a word, not a whole word and not a single letter. Common words like "the" or "cat" are often one token each. A longer or rarer word gets split, so "unbelievable" might become "un" + "believ" + "able".
The model never sees raw text, so both its input and its output are tokens. A token is not yet the mathematical form, though. Each token has an integer ID in the model's fixed vocabulary, and those IDs are what the model receives. Inside, the model turns each ID into a vector of numbers called an embedding, and that vector is the mathematical part.
So the order is text, then tokens, then token IDs, then vectors. It is worth keeping these four steps apart, because people often blur them.
Where tokenizers come from#
Each model developer builds its own tokenizer for each model family, and it is fixed before training starts. Nobody configures it by hand. It is learned from text: the algorithm scans a huge amount of data and keeps the character sequences that appear most often as single tokens, until the vocabulary reaches a set size.
Once the model is trained with that tokenizer, the two are locked together. The model only understands that tokenizer's IDs, so you cannot swap in one from another model.
This has two practical effects:
- The same sentence can be a different number of tokens on different models. Comparing prices per token across providers is not quite like for like.
- Some tokenizers are more efficient on code or non-English text, because more of that kind of text was in the data they learned from.
Why everything is measured in tokens#
The token is the model's unit of work, so everything is counted in tokens. Three things depend on it.
- Input: the model reads every token of your prompt. More input tokens means more computation, so providers charge per input token.
- Output: the model writes its answer one token at a time, and each new token is a full pass through the model. Output is slower to produce, which is why output tokens usually cost several times more than input tokens.
- Limits: the model can only hold a fixed number of token positions at once, so its maximum capacity is also counted in tokens. The context window note covers this limit.
This is also why a long answer feels slow. A reply of 500 output tokens means 500 passes through the model, one after another. How the model picks each token is the topic of next-token prediction.
Converting tokens to words#
For English text, a rough rule of thumb is that 1 token is about 4 characters, or about three quarters of a word.
| Tokens | Approximate words |
|---|---|
| 100 | 75 |
| 1,000 | 750, about a page and a half |
This is only an estimate. The real count depends on the tokenizer, and three kinds of text push it up:
- Code: symbols, indentation and variable names split into many small pieces.
- Numbers: a date like "2026-10-21" can be several tokens.
- Non-English text: many languages take more tokens for the same meaning.
For an exact count, use the provider's own tools, such as OpenAI's tiktoken library or Anthropic's token-counting endpoint. Only the model's own tokenizer gives the true number.
Why models miscount letters#
A classic question is why an LLM cannot count the r's in "strawberry". The model never sees the letters. It sees something like "str" + "aw" + "berry" as three token IDs.
Asking it to count letters is like asking someone how many letters are in three emoji they have only ever seen as pictures. The information is hidden inside the chunks.
The same root cause explains other odd weaknesses:
- Reversing or spelling words letter by letter. The model has to recall how each chunk is spelled instead of reading the letters.
- Arithmetic on long numbers. "123456" may split into uneven chunks like "123" + "456", which do not line up with place value.
- Rhymes and character-level wordplay, in some cases.
Newer models handle many of these better, often by spelling a word out letter by letter before answering. The underlying cause is the same, though: the model sees tokens, not characters.
Why sub-words?#
You might ask why tokenizers use sub-word pieces and not whole words or single characters. The table compares the three choices.
| Unit | Vocabulary | Sequence length | Main problem |
|---|---|---|---|
| Whole words | Millions of entries | Short | Cannot handle new words, typos or unseen names |
| Single characters | Tiny | About 4 times longer | More passes, more cost, less room in the context |
| Sub-words | Manageable | Short | No direct view of letters; some languages and code cost more |
Sub-words are the middle ground. Any word can be built from pieces, and sequences stay short. The price is the side effects in the earlier sections.
Short line#
A token is a sub-word piece with an integer ID. The model reads and writes only tokens, so cost, speed and limits are all counted in them, and it cannot see the letters inside a token.
Key takeaways#
- The order is text, then tokens, then token IDs, then embedding vectors.
- A tokenizer is learned from data, fixed before training, and tied to its model.
- Input, output and context limits are all counted in tokens. Output costs more because each token needs its own pass.
- English averages about 4 characters per token. Code, numbers and non-English text usually cost more.
- Models miscount letters because they see chunks, not characters.
- Sub-words balance vocabulary size against sequence length.