Learn · Module 02 · Classical machine learning · Lesson 2 of 8
Fitting a line
Linear regression is the simplest model there is, and the ideas in it (a loss, a fit, a prediction) carry all the way to large language models.
- Linear regression fits y = w·x + b to predict a number.
- The best line minimises mean squared error, which punishes big misses most.
- It's the baseline every more complex model should beat.
In plain words
Regression predicts a number: a price, a score, a delivery time. The simplest version draws a straight line through the data, y = w·x + b, where w is the slope and b is where the line crosses zero.
Which line is best? The one with the smallest mean squared error: for each point, take the gap between the real value and the line's prediction, square it, and average. Squaring makes big misses count much more than small ones.
Worked example: hours studied and test score
| hours (x) | score (y) |
|---|---|
| 1 | 3 |
| 2 | 5 |
| 3 | 6 |
| 4 | 8 |
- 1Find the averages
average x = 2.5 average y = 5.5
- 2Find how far each point is from the averages
x − 2.5: −1.5, −0.5, 0.5, 1.5 y − 5.5: −2.5, −0.5, 0.5, 2.5
- 3Slope = sum of (x gap × y gap) ÷ sum of (x gap squared)
This formula gives the line with the smallest squared error.
(3.75 + 0.25 + 0.25 + 3.75) ÷ (2.25 + 0.25 + 0.25 + 2.25) = 8 ÷ 5 = 1.6
- 4The line passes through the averages
b = 5.5 − 1.6 × 2.5 = 1.5 line: y = 1.6x + 1.5
- 5Check the fit
Predictions are 3.1, 4.7, 6.3 and 7.9. The misses are −0.1, 0.3, −0.3 and 0.1.
MSE = (0.01 + 0.09 + 0.09 + 0.01) ÷ 4 = 0.05
Now predict: someone who studies 5 hours should score about 1.6 × 5 + 1.5 = 9.5.
A colleague suggests the line y = 2x + 1 instead. Its mean squared error on the same data is 0.5. Which line fits better?
Where you'll see this
Real problems have many inputs, so the line becomes y = w₁x₁ + w₂x₂ + … + b, but the idea is identical. Linear regression is the baseline to beat: if a fancy model can't beat it, the fancy model isn't worth it.
The pattern of this lesson repeats in every model: choose a model shape, define a loss, find the numbers that make the loss smallest, then predict.
Common mistakes
- Reading a high R² as proof the model is right. It can still be biased, overfitted or fitted to a non-linear pattern.
- Forgetting that squared error is very sensitive to outliers; one extreme point can tilt the whole line.
- Interpreting coefficients as causal effects when features are correlated with each other or with something unmeasured.
Answer in your own words, out loud or on paper, before you open the model answer.
Explain linear regression and its loss function.
Linear regression predicts a number as a weighted sum of the features plus a bias. To fit it, we choose the weights that minimise mean squared error, the average of the squared differences between predictions and actual values. Squaring makes all errors positive and punishes big misses more, and it makes the maths neat: there's a closed-form solution, or we can use gradient descent for large data. It assumes the relationship is roughly linear and that errors are similar in size everywhere. Its weaknesses are sensitivity to outliers and correlated features, so in practice I'd check residual plots and often add regularisation, like ridge, to keep the weights stable.
- Why square the errors instead of taking their absolute value?
- When would linear regression be a bad choice?
Go further
- DocsLinear regressionGoogle Machine Learning Crash Course
- BookAn Introduction to Statistical Learning (free book)James, Witten, Hastie & Tibshirani