A token is the unit of text a language model reads and writes. It can be a whole word, part of a word, a single character, a punctuation mark or a space attached to a word. Before a model sees any text, a tool called a tokenizer cuts it into tokens and replaces each one with a number the model can process.
A large language model works entirely in tokens. It receives a sequence of token numbers, predicts the most likely next token, adds it to the sequence and repeats. The words you see in an answer are those tokens converted back into text.
How does a model split text into tokens?
Each model has a fixed vocabulary, a list of tens of thousands of text fragments with an ID number for each. The vocabulary is built once, before training, from a large sample of text. Most modern models use a method called byte pair encoding, which works roughly like this:
- Start with individual characters (or bytes) as the vocabulary.
- Count which pairs of neighbouring pieces appear most often in the sample text.
- Merge the most frequent pair into a new, longer piece and add it to the vocabulary.
- Repeat until the vocabulary reaches its target size.
In the finished vocabulary, common words such as "the" or "search" are single tokens, while rare words are built from several smaller pieces. A tokenizer might split "tokenization" into "token" and "ization", and an unusual brand name into three or four fragments. Leading spaces are usually part of the token, so " search" and "search" can have different IDs.
Different model families use different tokenizers, so the same sentence can produce a different number of tokens in two models.
Why do models use tokens instead of words or letters?
Tokens are a compromise between two worse options. A vocabulary of whole words would need millions of entries and would still fail on new words, typos, product codes and names. A vocabulary of single letters would handle anything, but every sentence would become a very long sequence, and the model would spend most of its capacity learning to spell.
Subword tokens keep sequences reasonably short while still covering any input. A word the model has never seen can always be written as a combination of smaller pieces it knows.
How many tokens is a piece of text?
The count depends on the tokenizer and on the kind of text. For ordinary English prose, a common rule of thumb is that one token is about four characters, or about three quarters of a word. Other kinds of text tokenize differently:
| Text | Typical tokenization |
|---|---|
| Common English words | One token each |
| Rare words, names, technical terms | Several tokens each |
| Long numbers, prices, dates | Split into short groups of digits |
| Languages with less training data | More tokens per word than English |
| HTML, code, repeated spaces | Many tokens for little readable content |
A page in German, Polish or Japanese therefore often uses more tokens than the same content in English. Markup and leftover navigation text also add tokens without adding meaning.
What do tokens limit?
Almost every practical limit of a language model is measured in tokens.
- Context window. This is the maximum number of tokens the model can consider at once, including the instructions, the conversation, any retrieved documents and the answer it is writing. Text beyond the limit is dropped or never sent.
- Output length. Answers are capped at a set number of tokens, which is why long responses sometimes stop mid-sentence.
- Cost. API providers charge per token, usually with separate rates for input and output tokens.
- Speed. The model produces one token per step, so a longer answer takes longer to generate.
Embeddings also start from tokens. In semantic search, a passage is tokenized and then turned into a list of numbers that represents its meaning, and embedding models have their own token limits per passage.
Why do tokens cause some odd model errors?
The model sees token IDs, so it has no direct view of the letters inside a token. When asked to count the letters in a word, reverse a string or check a rhyme, it is working from fragments it cannot inspect, and it can get these simple tasks wrong. Arithmetic on long numbers has a similar problem, since the digits may be grouped in ways that do not match place values.
Tokenization also explains why spelling matters for names. A brand written consistently is a stable sequence of tokens that the model can associate with facts it has seen. Several spellings of the same name split into different sequences, and the model may treat them as separate things.
What do tokens mean for a website?
AI search systems use retrieval-augmented generation: they fetch pages, split them into passages and pass a selection to the model. Every passage costs tokens, and the context window has room for only a limited number of them, so the system keeps a small set of short passages from each source and discards the rest.
Passages are usually cut by token count, often a few hundred tokens each. A fact that sits in one compact paragraph near a clear heading survives that cut intact. If the same fact is spread across a long introduction, a sidebar and a footnote, it may be split across chunks and lose its context. Pages heavy with boilerplate, scripts or repeated menus waste part of the budget on text that answers nothing.
The ranking step that picks which passages reach the model is covered in how AI search chooses which sources to cite. For a website, this means each answer-worthy statement should make sense by itself within a short span of text, written in plain words and with names spelled the same way everywhere.
