Learn · Module 05 · Tokens and embeddings · Lesson 2 of 4
Meaning as direction
A token number is just a label. An embedding turns that label into a list of numbers where similar meanings point the same way.
- An embedding is a list of numbers that places a token, or a whole text, in space.
- Similar meanings end up pointing in similar directions; cosine similarity measures that.
- The numbers are learned, not designed, and real ones have hundreds or thousands of dimensions.
In plain words
Token 4,821 and token 4,822 have nothing in common except being neighbours in a list. The model needs something better: a description of each token that it can do maths on. That description is an embedding, a list of numbers, also called a vector.
The idea is simple. Give every token a position in space. Put tokens that are used in similar ways close together, and tokens that aren't far apart. Now 'similar meaning' becomes something you can measure: how close are the two vectors?
Nobody writes these numbers by hand. They start random and are adjusted during training until words that appear in similar sentences end up near each other.
Worked example: four foods, three numbers
To see how it works, we'll cheat and choose the numbers ourselves. Each food gets three numbers: how hot it is served, how much of a drink it is, and how solid it is.
| hot | drink | solid | |
|---|---|---|---|
| chai | 0.9 | 0.9 | 0.1 |
| coffee | 0.8 | 0.9 | 0.0 |
| samosa | 0.7 | 0.0 | 0.9 |
| ice cream | 0.0 | 0.1 | 0.8 |
The standard way to compare two vectors is cosine similarity: it measures the angle between them. 1 means they point the same way, 0 means they have nothing in common.
- 1Multiply matching numbers and add them up
This is the dot product of chai and coffee.
0.9×0.8 + 0.9×0.9 + 0.1×0.0 = 1.53
- 2Measure each vector's length
Square each number, add, take the square root.
chai: √(0.81 + 0.81 + 0.01) = 1.277 coffee: √(0.64 + 0.81 + 0) = 1.204
- 3Divide the dot product by both lengths
The result is the cosine similarity.
1.53 ÷ (1.277 × 1.204) = 0.995
Do the same for the other pairs and the picture is clear.
Samosa is [0.7, 0.0, 0.9] and ice cream is [0.0, 0.1, 0.8]. Without calculating, which is true?
What it looks like in a real model
Real embeddings work the same way with far more numbers. OpenAI's text-embedding-3-small gives every piece of text 1,536 numbers. No single number means 'hot' or 'drink'; the meanings are spread across all of them, and the model found them on its own.
There are two places you'll meet embeddings. Inside a language model, the first layer turns each token into a vector. Separately, embedding models turn a whole sentence or document into one vector, which is what search and RAG systems use.
Common mistakes
- Treating token ids as meaningful numbers. Id 500 isn't 'close to' id 501 in any sense.
- Comparing embeddings from two different models; their spaces aren't aligned.
- Assuming high similarity means two texts agree. Opposites like 'love' and 'hate' can sit close together.
Answer in your own words, out loud or on paper, before you open the model answer.
What is an embedding, and why not just use token ids?
An embedding is a learned vector of numbers that represents a token or a piece of text. Token ids are just labels, so their values carry no meaning, and a one-hot encoding would treat every word as equally different from every other. Embeddings fix that: during training, the model adjusts the vectors so that items used in similar contexts end up pointing in similar directions. That gives us a space where distance means something, so we can compare meanings with cosine similarity, which powers semantic search, clustering and retrieval. The model's first layer is an embedding table: each token id simply looks up its row, and everything after works on those vectors.
- How does cosine similarity differ from the dot product?
- How would you find documents similar to a user's question?
Go further
- ArticleThe Illustrated Word2vecJay Alammar
- ArticleEmbeddings: What they are and why they matterSimon Willison
- PaperSentence-BERT: Sentence Embeddings using Siamese BERT-NetworksReimers & Gurevych, 2019