epochs

Learn · Module 16 · ML from scratch · Lesson 2 of 5

Logistic regression from scratch

Implementing logistic regression shows you understand the loss, the gradient and the training loop, in about fifteen lines.

4 minute read

In short
  • The gradient of log loss is Xᵀ(p − y) ÷ n for the weights and mean(p − y) for the bias.
  • Training is the loop: predict, compute gradient, step.
  • On separable data weights grow without bound; regularisation fixes it.

The idea

From the Classical ML module: score = X·w + b, probability = sigmoid(score), loss = average log loss. The gradient of that loss has a neat form: for the weights it's Xᵀ(p − y) ÷ n, and for the bias it's the average of (p − y). Then step downhill.

The code

Python, NumPy only
import numpy as np

def sigmoid(z):
    return 1 / (1 + np.exp(-z))

def train(X, y, lr=0.5, steps=1000):
    w = np.zeros(X.shape[1])
    b = 0.0
    for _ in range(steps):
        p = sigmoid(X @ w + b)
        grad_w = X.T @ (p - y) / len(y)   # gradient of the average log loss
        grad_b = (p - y).mean()
        w -= lr * grad_w
        b -= lr * grad_b
    return w, b

# hours studied, practice tests taken -> passed?
X = np.array([[1, 0], [2, 1], [3, 0], [3, 2], [5, 1], [6, 2]], dtype=float)
y = np.array([0, 0, 0, 1, 1, 1], dtype=float)
w, b = train(X, y)
p = sigmoid(X @ w + b)
print(np.round(w, 2).tolist(), round(float(b), 2))  # [2.07, 3.71] -10.31
print(np.round(p, 2).tolist())  # [0.0, 0.08, 0.02, 0.97, 0.98, 1.0]
print(((p > 0.5) == y).mean())  # 1.0
  1. 1
    Both weights are positive

    More hours and more practice tests both raise the chance of passing.

  2. 2
    Look at rows 3 and 4

    Both studied 3 hours. The one who took 2 practice tests is predicted to pass (0.97); the one who took none isn't (0.02).

  3. 3
    The gradient is p − y

    If the prediction is too high, the weights move down, and vice versa. That one line is the heart of the whole algorithm.

Your turn

This data can be perfectly separated. What happens to the weights if you train for a million steps?

Take it further

Add L2 regularisation (grad_w += λ × w). Explain why we use log loss instead of squared error. Handle numerical overflow in exp for very large scores. Extend to many classes with softmax.

Common mistakes

  • Using squared error instead of log loss.
  • Writing a naive sigmoid that overflows for large negative inputs.
  • Forgetting to divide the gradient by the number of examples, which ties the learning rate to dataset size.
Check yourself

Answer in your own words, out loud or on paper, before you open the model answer.

Implement logistic regression with gradient descent.

I'd write a sigmoid function, then a training loop. Start with weights at zero and bias at zero. Each iteration: compute the predicted probabilities as the sigmoid of X times w plus b. The gradient of the average log loss is neat: for the weights it's X transposed times the vector of predictions minus labels, divided by n, and for the bias it's the mean of predictions minus labels. Update the weights and bias by subtracting the learning rate times those gradients, and repeat for a fixed number of steps or until the loss stops improving. Then I'd mention extensions: L2 regularisation by adding lambda times w to the gradient, a numerically stable sigmoid, and checking the loss decreases.

More to think about
  • Derive the gradient of the log loss.
  • How would you add regularisation?

Go further

Was this lesson clear?