Training a model means nudging its numbers downhill. The derivative tells you which way is down.
4 minute read
In short
The derivative is the slope of the loss for one weight; the gradient is the list of slopes for all weights.
Gradient descent subtracts learning rate × slope from each weight, again and again.
Too large a learning rate overshoots and blows up; too small is slow.
In plain words
A loss is a number that says how wrong a model is. Training means changing the model's weights to make the loss smaller. The question is: which way should each weight move?
The derivative answers it. It's the slope of the loss: how much the loss changes when you nudge a weight up a tiny bit. If the slope is positive, moving the weight up makes things worse, so move it down. With many weights, the list of all their slopes is the gradient.
Gradient descent repeats one rule: new weight = old weight − learning rate × slope. The learning rate controls the step size.
Worked example: walking downhill
Take the loss (w − 3)². It's smallest at w = 3. Its slope is 2(w − 3). Start at w = 0 with a learning rate of 0.1.
step
w
loss
slope
next w
0
0.000
9.000
−6.000
0.600
1
0.600
5.760
−4.800
1.080
2
1.080
3.686
−3.840
1.464
3
1.464
2.359
−3.072
1.771
4
1.771
1.510
−2.458
2.017
5
2.017
0.966
−1.966
2.214
Each step moves w towards 3, and the loss falls. The steps shrink on their own, because the slope gets smaller as you approach the bottom.
When the step is too big
With a learning rate of 1.1, the same start goes 0 → 6.6 → −1.32 → 8.18 → −3.22. Every step overshoots the bottom by more than the last, and training blows up. This is why the learning rate is the first thing to lower when a loss suddenly shoots up.
With two or more inputs, you take one slope per input. For f(x, y) = x² + 3y at the point (2, 1), the slope for x is 2x = 4 and for y is 3, so the gradient is [4, 3].
Your turn
For the loss (w − 3)², what is the slope at w = 5, and which way should w move?
Where you'll see this
Training a neural network is this exact loop with millions or billions of weights. The method that computes all the slopes at once is backpropagation, which has its own lesson in the Neural networks module.
Common mistakes
Moving with the gradient instead of against it. The gradient points uphill; to reduce the loss you step the other way.
Picking a learning rate without checking the loss curve. Too big oscillates or diverges, too small barely moves.
Thinking gradient descent always finds the best possible solution. On non-convex losses it finds a good local one.
Check yourself
Answer in your own words, out loud or on paper, before you open the model answer.
Explain gradient descent in simple terms.
Gradient descent is how we train most models. The loss tells you how wrong the model is for its current weights. The gradient is the vector of partial derivatives of the loss with respect to each weight, and it points in the direction where the loss increases fastest. So we take a small step in the opposite direction: new weights equal old weights minus the learning rate times the gradient, and repeat. The learning rate sets the step size, which is a trade-off between speed and stability. In practice we compute the gradient on a mini-batch rather than the whole dataset, which is stochastic gradient descent: noisier, but much faster per step.
More to think about
What happens if the learning rate is too high?
What is a gradient, for a function of many inputs?