epochs

Learn · Module 02 · Classical machine learning · Lesson 2 of 8

Fitting a line

Linear regression is the simplest model there is, and the ideas in it (a loss, a fit, a prediction) carry all the way to large language models.

4 minute read

In short
  • Linear regression fits y = w·x + b to predict a number.
  • The best line minimises mean squared error, which punishes big misses most.
  • It's the baseline every more complex model should beat.

In plain words

Regression predicts a number: a price, a score, a delivery time. The simplest version draws a straight line through the data, y = w·x + b, where w is the slope and b is where the line crosses zero.

Which line is best? The one with the smallest mean squared error: for each point, take the gap between the real value and the line's prediction, square it, and average. Squaring makes big misses count much more than small ones.

Worked example: hours studied and test score

hours (x)score (y)
13
25
36
48
  1. 1
    Find the averages
    average x = 2.5    average y = 5.5
  2. 2
    Find how far each point is from the averages
    x − 2.5: −1.5, −0.5, 0.5, 1.5
    y − 5.5: −2.5, −0.5, 0.5, 2.5
  3. 3
    Slope = sum of (x gap × y gap) ÷ sum of (x gap squared)

    This formula gives the line with the smallest squared error.

    (3.75 + 0.25 + 0.25 + 3.75) ÷ (2.25 + 0.25 + 0.25 + 2.25)
    = 8 ÷ 5 = 1.6
  4. 4
    The line passes through the averages
    b = 5.5 − 1.6 × 2.5 = 1.5
    line: y = 1.6x + 1.5
  5. 5
    Check the fit

    Predictions are 3.1, 4.7, 6.3 and 7.9. The misses are −0.1, 0.3, −0.3 and 0.1.

    MSE = (0.01 + 0.09 + 0.09 + 0.01) ÷ 4 = 0.05

Now predict: someone who studies 5 hours should score about 1.6 × 5 + 1.5 = 9.5.

Your turn

A colleague suggests the line y = 2x + 1 instead. Its mean squared error on the same data is 0.5. Which line fits better?

Where you'll see this

Real problems have many inputs, so the line becomes y = w₁x₁ + w₂x₂ + … + b, but the idea is identical. Linear regression is the baseline to beat: if a fancy model can't beat it, the fancy model isn't worth it.

The pattern of this lesson repeats in every model: choose a model shape, define a loss, find the numbers that make the loss smallest, then predict.

Common mistakes

  • Reading a high R² as proof the model is right. It can still be biased, overfitted or fitted to a non-linear pattern.
  • Forgetting that squared error is very sensitive to outliers; one extreme point can tilt the whole line.
  • Interpreting coefficients as causal effects when features are correlated with each other or with something unmeasured.
Check yourself

Answer in your own words, out loud or on paper, before you open the model answer.

Explain linear regression and its loss function.

Linear regression predicts a number as a weighted sum of the features plus a bias. To fit it, we choose the weights that minimise mean squared error, the average of the squared differences between predictions and actual values. Squaring makes all errors positive and punishes big misses more, and it makes the maths neat: there's a closed-form solution, or we can use gradient descent for large data. It assumes the relationship is roughly linear and that errors are similar in size everywhere. Its weaknesses are sensitivity to outliers and correlated features, so in practice I'd check residual plots and often add regularisation, like ridge, to keep the weights stable.

More to think about
  • Why square the errors instead of taking their absolute value?
  • When would linear regression be a bad choice?

Go further

Was this lesson clear?