Learn · Module 05 · Tokens and embeddings · Lesson 1 of 4
How text becomes numbers
A model never sees your words. It sees a list of numbers. This lesson shows where those numbers come from, and why that changes what you pay.
- A tokenizer turns text into token numbers; the model only ever sees those.
- Byte-pair encoding builds the vocabulary by merging the most common neighbouring pair, over and over.
- Cost and context limits are counted in tokens, and the same text can be very different lengths in different languages.
In plain words
A language model does arithmetic, so text has to become numbers first. The piece that does this is the tokenizer. It cuts text into small pieces called tokens and swaps each one for its number in a fixed list, the vocabulary.
Why not one number per word? There are too many words, and new ones appear every day, so the model would keep meeting words it has no number for. Why not one number per letter? Then every sentence becomes very long, and a single letter carries almost no meaning.
Tokens sit in between. Common words get their own token. Rare words are built from smaller pieces, so nothing is ever unknown: token ization is two tokens.
Worked example: learning a vocabulary
The most common method is called byte-pair encoding. It starts with single characters and repeats one move: find the pair of neighbours that appears most often, and merge it into a new token. Here it is on four words, with how often each appears.
- 0Start with letters
Every word is split into single characters. The vocabulary is just a, c, h, i, t.
chai ×6chaichat ×4chathat ×3hatthat ×8that - 1Merge the most common pair: h + a
h followed by a appears 6 + 4 + 3 + 8 = 21 times, more than any other pair. It becomes one token, ha.
chai ×6chaichat ×4chathat ×3hatthat ×8that - 2Count again, merge again: ha + t
Now ha followed by t is the most common pair: 4 + 3 + 8 = 15 times. It becomes hat.
chai ×6chaichat ×4chathat ×3hatthat ×8that - 3One more: t + hat
t followed by hat appears 8 times, ahead of c + ha at 6. The whole word that is now a single token.
chai ×6chaichat ×4chathat ×3hatthat ×8that
After three merges the vocabulary has eight tokens: a, c, h, i, t, ha, hat and that. Notice what happened to chai. It was the second most common word, but it is still three tokens, because its pieces weren't the most common pairs. A tokenizer keeps frequent pieces, not frequent words.
A new word arrives: chat. Using the three merges in order, how is it split?
What it looks like in a real model
GPT-4o's tokenizer has a vocabulary of about 200,000 tokens, built the same way. Here is one sentence in three languages.
Same meaning, twice the tokens in Telugu. The likely reason: the text it learned from had far more English than Telugu, so it kept fewer long Telugu pieces and has to spell words out in small ones. Since you pay per token, and the context window is measured in tokens, the Telugu version costs twice as much and fills the window twice as fast.
One more detail: the space belongs to the token. In the English sentence the tokens are I am going and so on, each carrying its leading space.
Common mistakes
- Estimating cost by word count. Prices and limits are in tokens, and the ratio differs a lot by language.
- Forgetting that numbers and rare words split into many tokens, which hurts arithmetic and spelling tasks.
- Assuming every model uses the same tokenizer; counts differ between model families.
Answer in your own words, out loud or on paper, before you open the model answer.
Why do LLMs use subword tokens instead of words or characters?
Models work on integer ids, so text must be split into units first. Whole words would need a huge vocabulary and still fail on new words, names and typos. Characters cover everything but make sequences very long, which costs more compute because attention grows with length. Subword tokens, learned with methods like byte-pair encoding, balance the two: common words become single tokens, while rare words split into familiar pieces, so nothing is ever unknown. The choice has practical effects: you pay per token, context limits are in tokens, some languages use far more tokens for the same meaning, and models struggle with tasks like counting letters because they never see individual characters.
- Walk me through how byte-pair encoding builds a vocabulary.
- Our users write in Hindi and our costs doubled. Why might that be?
Go further
- VideoLet's build the GPT TokenizerAndrej Karpathy
- DocsByte-Pair Encoding tokenizationHugging Face LLM Course
- PaperNeural Machine Translation of Rare Words with Subword UnitsSennrich et al., 2015