epochs

Learn · Module 05 · Tokens and embeddings · Lesson 4 of 4

Telling the model about order

'Dog bites man' and 'man bites dog' use the same tokens. Without help, a transformer can't tell them apart.

4 minute read

In short
  • Attention treats tokens as a set, so order has to be added as extra information.
  • The classic method adds a position vector, made of sine and cosine waves, to each token's embedding.
  • Most modern LLMs use rotary position embeddings, which encode position inside attention instead.

This lesson is part of Pro

The first two lessons of every module are free. Pro opens every lesson and every project.

See Pro plansFrom ₹119 a month. No auto-renewal.