A model that memorises its training data looks perfect until it meets new data. Here is how to spot it and stop it.
4 minute read
In short
Overfitting means great training error and poor test error; underfitting means both are poor.
Always measure on data the model didn't train on.
Regularisation penalises large weights, trading a little training error for much better generalisation.
In plain words
Overfitting is learning the noise instead of the pattern. The model does brilliantly on the data it trained on and badly on anything new. Underfitting is the opposite: the model is too simple to capture the pattern at all.
You catch both by holding some data back. The training set is used to fit; the test set (or validation set) is only used to check. A big gap between training error and test error means overfitting.
Worked example: three curves through the same points
Twelve noisy points are drawn from a wave. Fit three polynomials of increasing complexity, then measure error on the 12 training points and on 200 new points.
model
training error
test error
verdict
straight line (degree 1)
0.293
0.325
underfits
cubic (degree 3)
0.033
0.125
about right
degree 9
0.016
13.119
overfits badly
The degree-9 curve has the lowest training error but bends wildly between the points, so its test error is 100 times worse.
Worked example: regularisation
Regularisation adds a penalty for large weights to the loss, so the model prefers simpler fits. With L2 regularisation (also called ridge), the penalty is λ times the sum of squared weights. Keep the degree-9 model but add the penalty.
penalty λ
training error
test error
largest weight
0.0001
0.026
0.106
14.3
0.01
0.078
0.162
4.7
1
0.329
0.344
0.6
A small penalty tames the same flexible model: test error falls from 13.1 to 0.106. Too large a penalty flattens it into underfitting again. λ is chosen by checking the validation set.
Your turn
A model has training loss 0.01 and test loss 2.5. What's most likely happening?
Where you'll see this
Neural networks use the same ideas with different names: weight decay is L2 regularisation, dropout randomly switches off neurons during training, and early stopping halts training when validation loss starts rising. The Reading a loss curve lesson shows that last one.
Common mistakes
Tuning on the test set. Once you've used it to make choices, it no longer measures real performance.
Splitting time-ordered data randomly, which lets the model peek at the future and hides overfitting.
Only adding regularisation, when more data or simpler features would fix the problem better.
Check yourself
Answer in your own words, out loud or on paper, before you open the model answer.
How do you detect overfitting?
The main signal is a gap between training and validation performance: the model does very well on data it has seen and much worse on data it hasn't. I'd split the data into training, validation and test sets, or use cross-validation when data is small, and watch both scores as model complexity or training time grows. If training error keeps falling while validation error flattens or rises, that's overfitting. Fixes include more data, simpler models, regularisation, early stopping, dropout for neural networks, and removing features that leak information. I also keep the test set untouched until the end, so the final number is honest.
More to think about
Explain L1 versus L2 regularisation.
Your model does well in testing but badly in production. What could be happening?