'Dog bites man' and 'man bites dog' use the same tokens. Without help, a transformer can't tell them apart.
4 minute read
In short
Attention treats tokens as a set, so order has to be added as extra information.
The classic method adds a position vector, made of sine and cosine waves, to each token's embedding.
Most modern LLMs use rotary position embeddings, which encode position inside attention instead.
In plain words
The core of a transformer, attention, looks at all tokens at once and treats them as a set. It has no built-in sense of first, second or last. Shuffle the tokens and, without extra information, it computes the same thing.
The fix is to give each token two pieces of information: what it is (its embedding) and where it is (its position). The simplest way is to make a second vector for the position and add it to the embedding.
Worked example: positions made of waves
The original transformer built position vectors from sine and cosine waves of different speeds. With four numbers per position it uses a fast wave pair and a slow wave pair.
position
sin(p)
cos(p)
sin(p/100)
cos(p/100)
0
0.000
1.000
0.000
1.000
1
0.841
0.540
0.010
1.000
2
0.909
−0.416
0.020
1.000
The fast pair changes a lot between neighbours, so the model can tell position 1 from position 2. The slow pair barely moves, so it helps tell position 5 from position 500. Together, every position gets a unique pattern.
1
Take the token's embedding
Say the token 'dog' has this (made-up) embedding.
[0.200, 0.500, 0.100, 0.300]
2
Add the vector for its position
If 'dog' is at position 1, add row 1 of the table, number by number.
The same token at position 0 would become [0.200, 1.500, 0.100, 1.300]. Now the model can tell where 'dog' is.
position 0 → [0.200, 1.500, 0.100, 1.300]
Your turn
A transformer has no position information at all. It reads 'dog bites man' and then 'man bites dog'. What does it see?
What real models do today
Later models learned a position vector for each slot instead of using fixed waves. Most current LLMs go one step further with rotary position embeddings (RoPE): instead of adding a vector at the input, they rotate the vectors inside attention by an angle that depends on position.
What matters for you is the idea that survives all versions: attention compares tokens by content, and position information is what lets it also care about distance and order.
Common mistakes
Saying attention 'naturally' knows word order. Without positions it treats the input as a bag of tokens.
Assuming a model trained on short contexts will work well on much longer ones without adjustment.
Confusing positional encodings with the token embeddings themselves.
Check yourself
Answer in your own words, out loud or on paper, before you open the model answer.
Why does a transformer need positional encodings?
Self-attention compares every token with every other token using dot products, and nothing in that calculation depends on where tokens are. If you shuffle the input, each token gets the same attention scores, just rearranged. So without extra information, 'dog bites man' and 'man bites dog' look the same. Positional information fixes that. The original transformer added fixed sine and cosine patterns to the embeddings; others learn a vector per position. Most modern LLMs use rotary embeddings, which rotate queries and keys by an angle that depends on position, so attention scores naturally reflect relative distance. How positions are encoded also affects how well a model handles longer contexts than it was trained on.
More to think about
What's the difference between absolute and relative position information?
What happens if you feed a model a sequence longer than it was trained on?