Learn · Module 06 · Transformers · Lesson 1 of 6
Attention, step by step
Attention is how a token gathers information from the other tokens. It is four small steps, and you can do them by hand.
- Each token has a query, a key and a value; attention matches queries against keys.
- The four steps are score, scale, softmax and mix.
- The output for a token is a weighted blend of the values of the tokens it attended to.
In plain words
Read this sentence: Ravi dropped the glass because it was wet. To understand 'it', you look back and decide it means the glass, not Ravi. Attention is the part of a transformer that does this looking back.
Every token gets three vectors. A query says what this token is looking for. A key says what this token offers to others. A value is the information it hands over if chosen. A token compares its query with every key, and takes more of the values whose keys match best.
Worked example: what does 'it' look at?
Keep three tokens: Ravi, glass and it. Use two numbers per vector so the arithmetic stays small. The query for 'it' is [1, 3].
| token | key | value |
|---|---|---|
| Ravi | [2, 0] | [1, 0] |
| glass | [1, 2] | [0, 1] |
| it | [0, 1] | [0.5, 0.5] |
- 1Score: dot product of the query with each key
A higher score means a better match.
Ravi: 1×2 + 3×0 = 2 glass: 1×1 + 3×2 = 7 it: 1×0 + 3×1 = 3
- 2Scale: divide by the square root of the vector size
The vectors have 2 numbers, so divide by √2 = 1.414. This keeps scores from growing too large in big models.
Ravi 1.414 glass 4.950 it 2.121
- 3Softmax: turn scores into weights that add up to 1
Raise e to each score, then divide by the total.
Ravi 0.027 glass 0.919 it 0.054
- 4Mix: add up the values, using the weights
The result is the new vector for 'it'.
0.027×[1, 0] + 0.919×[0, 1] + 0.054×[0.5, 0.5] = [0.054, 0.946]
The new vector for 'it' is [0.054, 0.946], almost identical to the value of 'glass', [0, 1]. The token 'it' now carries the information that it refers to the glass. That is everything attention does.
Suppose the key for Ravi were [4, 2] instead of [2, 0]. Its score against the query [1, 3] becomes 1×4 + 3×2 = 10. What happens?
What it looks like in a real model
A real model does this for every token at the same time, with vectors of hundreds or thousands of numbers. The queries, keys and values aren't chosen by hand: each is made by multiplying the token's vector by a matrix that the model learns.
It also runs several attentions side by side, called heads. One head might track what a pronoun refers to, another which adjective belongs to which noun. Their results are joined together.
In a model that generates text, a token may only look at tokens before it, never after. This rule is called the causal mask, and it is what lets the model be trained to predict the next token.
Common mistakes
- Forgetting the division by √d, which keeps scores from growing with dimension and saturating the softmax.
- Mixing up roles: queries ask, keys are matched against, values are what's actually mixed into the output.
- Ignoring that attention cost grows with the square of the sequence length.
Answer in your own words, out loud or on paper, before you open the model answer.
Explain self-attention as if I'd never seen it.
Self-attention lets each word gather information from the other words in the sentence. Each token makes three vectors: a query, which is what it's looking for; a key, which is what it offers; and a value, which is the information it passes on. For one token, we compare its query with every key using dot products, scale by the square root of the dimension, and apply softmax to turn the scores into weights that add up to one. The output is the weighted average of the values. So in 'Ravi dropped the glass and it broke', the token 'it' can put most of its weight on 'glass'. Multiple heads do this in parallel, each learning different kinds of relationships.
- Why do we divide by the square root of the dimension?
- What is multi-head attention for?
Go further
- ArticleThe Illustrated TransformerJay Alammar
- VideoAttention in transformers, step by step3Blue1Brown
- PaperAttention Is All You NeedVaswani et al., 2017