epochs

Learn · Module 16 · ML from scratch · Lesson 3 of 5

Attention in NumPy

Writing attention yourself is the clearest proof that you understand transformers. It takes a dozen lines.

4 minute read

In short
  • Attention is three lines: scores, softmax, weighted sum of values.
  • The causal mask sets future scores to −∞ so their weights become 0.
  • Subtract the max inside softmax for numerical safety.

This lesson is part of Pro

The first two lessons of every module are free. Pro opens every lesson and every project.

See Pro plansFrom ₹119 a month. No auto-renewal.