epochs

Learn · Module 05 · Tokens and embeddings · Lesson 2 of 4

Meaning as direction

A token number is just a label. An embedding turns that label into a list of numbers where similar meanings point the same way.

5 minute read

In short
  • An embedding is a list of numbers that places a token, or a whole text, in space.
  • Similar meanings end up pointing in similar directions; cosine similarity measures that.
  • The numbers are learned, not designed, and real ones have hundreds or thousands of dimensions.

In plain words

Token 4,821 and token 4,822 have nothing in common except being neighbours in a list. The model needs something better: a description of each token that it can do maths on. That description is an embedding, a list of numbers, also called a vector.

The idea is simple. Give every token a position in space. Put tokens that are used in similar ways close together, and tokens that aren't far apart. Now 'similar meaning' becomes something you can measure: how close are the two vectors?

Nobody writes these numbers by hand. They start random and are adjusted during training until words that appear in similar sentences end up near each other.

Worked example: four foods, three numbers

To see how it works, we'll cheat and choose the numbers ourselves. Each food gets three numbers: how hot it is served, how much of a drink it is, and how solid it is.

hotdrinksolid
chai0.90.90.1
coffee0.80.90.0
samosa0.70.00.9
ice cream0.00.10.8

The standard way to compare two vectors is cosine similarity: it measures the angle between them. 1 means they point the same way, 0 means they have nothing in common.

  1. 1
    Multiply matching numbers and add them up

    This is the dot product of chai and coffee.

    0.9×0.8 + 0.9×0.9 + 0.1×0.0 = 1.53
  2. 2
    Measure each vector's length

    Square each number, add, take the square root.

    chai: √(0.81 + 0.81 + 0.01) = 1.277
    coffee: √(0.64 + 0.81 + 0) = 1.204
  3. 3
    Divide the dot product by both lengths

    The result is the cosine similarity.

    1.53 ÷ (1.277 × 1.204) = 0.995

Do the same for the other pairs and the picture is clear.

chai and coffee0.995
both hot drinks
chai and samosa0.495
both served hot, but one is a drink
chai and ice cream0.165
almost nothing in common
Your turn

Samosa is [0.7, 0.0, 0.9] and ice cream is [0.0, 0.1, 0.8]. Without calculating, which is true?

What it looks like in a real model

Real embeddings work the same way with far more numbers. OpenAI's text-embedding-3-small gives every piece of text 1,536 numbers. No single number means 'hot' or 'drink'; the meanings are spread across all of them, and the model found them on its own.

There are two places you'll meet embeddings. Inside a language model, the first layer turns each token into a vector. Separately, embedding models turn a whole sentence or document into one vector, which is what search and RAG systems use.

Common mistakes

  • Treating token ids as meaningful numbers. Id 500 isn't 'close to' id 501 in any sense.
  • Comparing embeddings from two different models; their spaces aren't aligned.
  • Assuming high similarity means two texts agree. Opposites like 'love' and 'hate' can sit close together.
Check yourself

Answer in your own words, out loud or on paper, before you open the model answer.

What is an embedding, and why not just use token ids?

An embedding is a learned vector of numbers that represents a token or a piece of text. Token ids are just labels, so their values carry no meaning, and a one-hot encoding would treat every word as equally different from every other. Embeddings fix that: during training, the model adjusts the vectors so that items used in similar contexts end up pointing in similar directions. That gives us a space where distance means something, so we can compare meanings with cosine similarity, which powers semantic search, clustering and retrieval. The model's first layer is an embedding table: each token id simply looks up its row, and everything after works on those vectors.

More to think about
  • How does cosine similarity differ from the dot product?
  • How would you find documents similar to a user's question?

Go further

Was this lesson clear?