Learn · Module 16 · ML from scratch · Lesson 3 of 5
Attention in NumPy
Writing attention yourself is the clearest proof that you understand transformers. It takes a dozen lines.
4 minute read
- Attention is three lines: scores, softmax, weighted sum of values.
- The causal mask sets future scores to −∞ so their weights become 0.
- Subtract the max inside softmax for numerical safety.
The idea
From the Transformers module: scores = Q Kᵀ ÷ √d, weights = softmax of each row, output = weights × V. For text generation, add a causal mask so each token can't see tokens after it.
The code
import numpy as np
def softmax(x, axis=-1):
x = x - x.max(axis=axis, keepdims=True) # for numerical safety
e = np.exp(x)
return e / e.sum(axis=axis, keepdims=True)
def attention(Q, K, V, causal=False):
d = Q.shape[-1]
scores = Q @ K.T / np.sqrt(d) # (n, n)
if causal:
n = scores.shape[0]
scores = np.where(np.tril(np.ones((n, n))) == 1, scores, -np.inf)
weights = softmax(scores) # each row sums to 1
return weights @ V, weights
# The three tokens from the Attention lesson: Ravi, glass, it
K = np.array([[2, 0], [1, 2], [0, 1]], dtype=float)
V = np.array([[1, 0], [0, 1], [0.5, 0.5]])
q_it = np.array([[1, 3]], dtype=float)
out, w = attention(q_it, K, V)
print(np.round(w, 3).tolist(), np.round(out, 3).tolist())
# [[0.027, 0.919, 0.054]] [[0.054, 0.946]]That's exactly the hand calculation from the Attention lesson: 'it' puts 91.9% of its attention on 'glass'.
Q = np.array([[1, 0], [0, 1], [1, 3]], dtype=float) out, w = attention(Q, K, V, causal=True) print(np.round(w, 3).tolist()) # [[1.0, 0.0, 0.0], # [0.196, 0.804, 0.0], # [0.027, 0.919, 0.054]]
- 1Row 1 can only see token 1
So all its weight is on itself.
- 2Row 2 sees tokens 1 and 2
The third column is exactly zero: the mask set its score to −∞, and e^−∞ = 0.
- 3Row 3 sees everything before and including itself
Same as the unmasked result for 'it'.
Why subtract the row maximum inside softmax?
Take it further
Add batches and multiple heads (reshape to (batch, heads, n, d)). Explain the cost: the score matrix is n × n, so memory and time grow with the square of the sequence length. Add the learned projections Q = X W_q, K = X W_k, V = X W_v.
Common mistakes
- Forgetting to scale by the square root of the dimension.
- Applying softmax over the wrong axis, so columns sum to one instead of rows.
- Using a large negative number for masking but then adding it after the softmax instead of before.
Answer in your own words, out loud or on paper, before you open the model answer.
Implement scaled dot-product attention in NumPy.
I'd write a stable softmax first, subtracting the maximum of each row before exponentiating. Then attention takes Q, K and V. Compute scores as Q times K transposed, divided by the square root of the key dimension, giving an n-by-n matrix where row i says how much token i attends to each token. For a causal mask, set every score above the diagonal to minus infinity, using np.triu or np.tril, so after the softmax those weights are exactly zero. Apply softmax along the last axis so each row sums to one, and multiply the weights by V to get the output. I'd mention the n-squared memory and time cost, and how multi-head attention reshapes into heads.
- Add a causal mask to your implementation.
- What's the time and memory complexity of attention?
Go further
- VideoLet's build GPT: from scratch, in code, spelled outAndrej Karpathy
- PaperAttention Is All You NeedVaswani et al., 2017
- ArticleThe Illustrated TransformerJay Alammar
This lesson is part of Pro
The first two lessons of every module are free. Pro opens every lesson and every project.
See Pro plansFrom ₹119 a month. No auto-renewal.