Learn · Module 16 · ML from scratch · Lesson 2 of 5
Logistic regression from scratch
Implementing logistic regression shows you understand the loss, the gradient and the training loop, in about fifteen lines.
- The gradient of log loss is Xᵀ(p − y) ÷ n for the weights and mean(p − y) for the bias.
- Training is the loop: predict, compute gradient, step.
- On separable data weights grow without bound; regularisation fixes it.
The idea
From the Classical ML module: score = X·w + b, probability = sigmoid(score), loss = average log loss. The gradient of that loss has a neat form: for the weights it's Xᵀ(p − y) ÷ n, and for the bias it's the average of (p − y). Then step downhill.
The code
import numpy as np
def sigmoid(z):
return 1 / (1 + np.exp(-z))
def train(X, y, lr=0.5, steps=1000):
w = np.zeros(X.shape[1])
b = 0.0
for _ in range(steps):
p = sigmoid(X @ w + b)
grad_w = X.T @ (p - y) / len(y) # gradient of the average log loss
grad_b = (p - y).mean()
w -= lr * grad_w
b -= lr * grad_b
return w, b
# hours studied, practice tests taken -> passed?
X = np.array([[1, 0], [2, 1], [3, 0], [3, 2], [5, 1], [6, 2]], dtype=float)
y = np.array([0, 0, 0, 1, 1, 1], dtype=float)
w, b = train(X, y)
p = sigmoid(X @ w + b)
print(np.round(w, 2).tolist(), round(float(b), 2)) # [2.07, 3.71] -10.31
print(np.round(p, 2).tolist()) # [0.0, 0.08, 0.02, 0.97, 0.98, 1.0]
print(((p > 0.5) == y).mean()) # 1.0- 1Both weights are positive
More hours and more practice tests both raise the chance of passing.
- 2Look at rows 3 and 4
Both studied 3 hours. The one who took 2 practice tests is predicted to pass (0.97); the one who took none isn't (0.02).
- 3The gradient is p − y
If the prediction is too high, the weights move down, and vice versa. That one line is the heart of the whole algorithm.
This data can be perfectly separated. What happens to the weights if you train for a million steps?
Take it further
Add L2 regularisation (grad_w += λ × w). Explain why we use log loss instead of squared error. Handle numerical overflow in exp for very large scores. Extend to many classes with softmax.
Common mistakes
- Using squared error instead of log loss.
- Writing a naive sigmoid that overflows for large negative inputs.
- Forgetting to divide the gradient by the number of examples, which ties the learning rate to dataset size.
Answer in your own words, out loud or on paper, before you open the model answer.
Implement logistic regression with gradient descent.
I'd write a sigmoid function, then a training loop. Start with weights at zero and bias at zero. Each iteration: compute the predicted probabilities as the sigmoid of X times w plus b. The gradient of the average log loss is neat: for the weights it's X transposed times the vector of predictions minus labels, divided by n, and for the bias it's the mean of predictions minus labels. Update the weights and bias by subtracting the learning rate times those gradients, and repeat for a fixed number of steps or until the loss stops improving. Then I'd mention extensions: L2 regularisation by adding lambda times w to the gradient, a numerically stable sigmoid, and checking the loss decreases.
- Derive the gradient of the log loss.
- How would you add regularisation?
Go further
- BookCS229 lecture notes (logistic regression)Stanford
- DocsLogistic regressionGoogle Machine Learning Crash Course