epochs

Learn · Module 03 · Neural networks · Lesson 2 of 7

Layers and activations

One neuron draws a straight line. Two layers with an activation in between can draw almost any shape.

3 minute read

In short
  • Layers feed into each other; hidden layers sit between input and output.
  • Without activations, any stack of layers collapses into one linear layer.
  • Two layers with ReLU can solve XOR, which no single neuron can.

In plain words

Stack neurons into layers: the outputs of one layer become the inputs of the next. Layers between input and output are called hidden layers.

The activation function is what makes stacking worthwhile. Without it, two layers multiply into a single matrix, which is still a straight-line model. ReLU's 'zero out the negatives' is a bend, and enough bends can approximate almost any shape.

Worked example: solving XOR

XOR outputs 1 when exactly one of two inputs is 1. No single straight line separates its answers, so one neuron can't learn it. Two layers can. The hidden layer has two neurons with weights [1, 1] each and biases 0 and −1, then ReLU. The output neuron has weights [1, −2].

inputbefore ReLUhidden (after ReLU)outputXOR
[0, 0][0, −1][0, 0]00
[0, 1][1, 0][1, 0]11
[1, 0][1, 0][1, 0]11
[1, 1][2, 1][2, 1]2 − 2 = 00

The second hidden neuron only switches on when both inputs are 1, and the output weight of −2 cancels the first neuron in exactly that case. ReLU zeroing the −1 in the first row is what makes it work.

Your turn

Remove the ReLU from the hidden layer of this network. What happens?

Where you'll see this

Real networks use the same recipe at scale: matrix multiply, activation, repeat. Inside every transformer block is exactly this kind of two-layer network, usually four times wider than the token vector.

Common mistakes

  • Stacking layers without non-linear activations and expecting more power; it's still a linear model.
  • Assuming deeper is always better. Without good initialisation and normalisation, deep networks can fail to train.
  • Confusing width (neurons per layer) with depth (number of layers).
Check yourself

Answer in your own words, out loud or on paper, before you open the model answer.

Why can't a single-layer network solve XOR?

A single-layer network, meaning one neuron with a threshold, can only draw one straight line through the input space. XOR outputs 1 when exactly one input is 1, so the positive points are at opposite corners of a square, and no single straight line can separate them from the other two corners. Adding a hidden layer fixes this: two hidden neurons can each draw a line, for example one detecting 'at least one is on' and another 'both are on', and the output neuron combines them as 'first but not second'. That's the key idea behind depth: each layer builds new features from the previous ones, and the non-linear activations are what make that combination more powerful than one line.

More to think about
  • What happens if you remove all activation functions from a deep network?
  • How do you choose the number and width of layers?

Go further

Was this lesson clear?