Learn · Module 06 · Transformers · Lesson 2 of 6
Inside a transformer block
A transformer is the same block repeated many times. Each block has two jobs: let tokens talk, then let each token think.
- A transformer is one block repeated: attention so tokens share information, then an MLP so each token processes it.
- Residual connections add each step's result to its input, so blocks adjust vectors instead of replacing them.
- Most of a block's parameters sit in the MLP.
In plain words
A language model is a stack of identical blocks. A vector for every token goes in at the bottom, and a better vector for every token comes out at the top. Each block does two things.
First, attention: tokens exchange information, as in the last lesson. Second, a small feed-forward network, often called the MLP: each token's vector is processed on its own, with no view of the others. People describe it as tokens talking, then thinking.
Two details hold it together. A residual connection adds each step's result to its input instead of replacing it, so information is never lost on the way up. And normalisation rescales the numbers so they stay in a healthy range across many layers.
Worked example: one token through one block
Follow a single token's vector, [1.0, 2.0], through a block. The numbers for the attention and MLP outputs are made up; what matters is how they combine.
- 1Attention produces an update
After looking at the other tokens, attention outputs a small change.
attention output: [0.3, −0.1]
- 2Add it to the input (residual)
The token keeps what it had, plus what it learned from others.
[1.0, 2.0] + [0.3, −0.1] = [1.3, 1.9]
- 3The MLP produces another update
It works on this token alone.
MLP output: [−0.2, 0.4]
- 4Add again (residual)
This is the vector handed to the next block.
[1.3, 1.9] + [−0.2, 0.4] = [1.1, 2.3]
Every block only adjusts the vector. After many blocks, those small adjustments add up to a vector that knows a lot about the token's role in the sentence.
Worked example: counting the parameters
The numbers a model learns are its parameters. Count them for a tiny block where each token vector has 4 numbers and the MLP's middle layer has 16.
- 1Attention: four matrices
One each for queries, keys and values, and one for the output. Each is 4×4 plus 4 bias numbers.
4 × (4×4 + 4) = 80
- 2MLP: up to 16, back down to 4
Two layers, each with weights and biases.
(4×16 + 16) + (16×4 + 4) = 148
- 3Two normalisation layers
Each has a scale and a shift per number.
2 × (4 + 4) = 16
- 4Total for the block
80 + 148 + 16 = 244
In this tiny block, which part holds the most parameters?
What it looks like in a real model
GPT-2 small, a model you can run on a laptop, has 12 blocks, 768 numbers per token vector and 12 attention heads, for about 124 million parameters. Today's large models use the same block with more layers and much wider vectors, reaching many billions.
After the last block, one final layer turns the top vector into a score for every token in the vocabulary. Choosing the next token from those scores is the next lesson.
Common mistakes
- Forgetting the residual connections, which are what make very deep stacks trainable.
- Leaving out the feed-forward layer; it holds most of the parameters and much of the stored knowledge.
- Mixing up encoder and decoder blocks; GPT-style models use causal masking so tokens can't see the future.
Answer in your own words, out loud or on paper, before you open the model answer.
Walk me through a transformer block.
A block has two main parts, each wrapped with normalisation and a residual connection. First, multi-head self-attention: each token mixes in information from other tokens, with a causal mask in decoder models so it only sees earlier ones. The result is added back to the input. Second, a feed-forward network applied to each token separately, usually expanding to about four times the width and back with a non-linearity in between; that's added back too. Modern models normalise before each part, which trains more stably. Attention moves information between positions, the feed-forward layer processes it within each position, and stacking dozens of these blocks builds up increasingly abstract representations.
- Why do transformers use residual connections?
- Where are most of a transformer's parameters?
Go further
- ArticleThe Illustrated TransformerJay Alammar
- VideoLet's build GPT: from scratch, in code, spelled outAndrej Karpathy
- PaperAttention Is All You NeedVaswani et al., 2017