Backpropagation is how a network learns which weights to blame for a mistake. It's the chain rule, applied carefully.
4 minute read
In short
Backpropagation computes every weight's slope in one backward pass using the chain rule.
Each weight's slope is the product of the slopes along its path to the loss.
Frameworks do it automatically; understanding it helps you debug training.
In plain words
Training needs the slope of the loss for every weight. Backpropagation finds them all in one backward pass. It starts at the loss and works back through the network, multiplying slopes along the way. This is the chain rule: if a weight changes z, z changes the prediction, and the prediction changes the loss, then the weight's effect on the loss is all three slopes multiplied.
Worked example: one neuron, one update
Input x = 2, target y = 1, weight w = 0.5, bias b = 0. The neuron uses sigmoid, and the loss is (p − y)².
1
Forward pass
Compute the prediction and the loss.
z = 0.5 × 2 + 0 = 1.0
p = sigmoid(1.0) = 0.7311
loss = (0.7311 − 1)² = 0.0723
2
Slope of the loss with respect to p
2(p − y) = 2 × (−0.2689) = −0.5379
3
Slope of p with respect to z
For sigmoid this is p(1 − p).
0.7311 × 0.2689 = 0.1966
4
Slope of z with respect to w and b
z = wx + b, so changing w changes z by x, and changing b changes z by 1.
dz/dw = 2 dz/db = 1
5
Multiply along the chain
Negative slopes mean increasing w and b will lower the loss.
w = 0.5 − 0.1 × (−0.2115) = 0.5212
b = 0 − 0.1 × (−0.1058) = 0.0106
7
Check: forward again
The prediction moved towards 1 and the loss went down.
p = 0.7413 loss = 0.0669 (was 0.0723)
Your turn
Same neuron, but the input x is 0. What is the slope of the loss for the weight w?
Where you'll see this
In a deep network the chain is just longer: slopes are passed back layer by layer, and each layer reuses the work of the one above. That reuse is why a backward pass costs only about twice a forward pass, no matter how many weights there are.
Nobody does this by hand in practice. PyTorch and JAX record the forward pass and run backpropagation for you, which is called automatic differentiation. Knowing what it does helps you debug when gradients vanish, explode or come out as zero.
Common mistakes
Describing backprop as the learning algorithm itself. It computes gradients; the optimiser uses them to update weights.
Forgetting that the forward pass values must be stored, which is why training uses much more memory than inference.
Hand-waving the chain rule instead of showing how one layer's gradient feeds the previous one.
Check yourself
Answer in your own words, out loud or on paper, before you open the model answer.
Explain backpropagation in your own words.
Backpropagation is an efficient way to compute how much each weight contributed to the loss. First, the forward pass runs the input through the network, keeping each layer's intermediate values, and computes the loss. Then we go backwards. Using the chain rule, the gradient of the loss with respect to a layer's output is turned into gradients for that layer's weights and for its input, and the input gradient is passed back to the layer before. Because each layer reuses the gradient from the layer above, we get every weight's gradient in roughly the cost of one more forward pass. An optimiser like SGD or Adam then uses those gradients to update the weights.
More to think about
Why is a backward pass only about twice as expensive as a forward pass?