Learn · Module 01 · Python and maths toolkit · Lesson 2 of 5
Matrices as transformations
A matrix is a grid of numbers, but the useful way to think of it is as a machine that turns one vector into another.
- Multiplying a matrix by a vector takes the dot product of each row with the vector.
- A matrix transforms vectors: stretch, rotate, mix or change size.
- Shapes must line up: an (m × n) matrix takes n numbers in and gives m out.
In plain words
A matrix is a grid of numbers with rows and columns. Multiplying a matrix by a vector means taking the dot product of each row with the vector. Each row gives one number of the result.
That makes a matrix a transformation. Feed in a vector, get a new one: stretched, rotated, mixed, or turned into a different size. Every layer of a neural network is a matrix doing exactly this.
Worked example: stretch, then rotate
- 1A stretch
This matrix doubles the first number and leaves the second alone.
S = [[2, 0], [0, 1]] S × [1, 1] = [2×1 + 0×1, 0×1 + 1×1] = [2, 1] - 2A rotation by 90°
This matrix turns any vector a quarter turn anticlockwise.
R = [[0, −1], [1, 0]] R × [1, 1] = [−1, 1] - 3Order matters
Stretch then rotate is not the same as rotate then stretch.
R × (S × [1, 1]) = [−1, 2] S × (R × [1, 1]) = [−2, 1]
Worked example: changing size
A matrix doesn't have to be square. A matrix with 2 rows and 3 columns takes a 3-number vector and returns a 2-number vector.
W = [[1, 2, 0],
[0, 1, −1]] 2 rows × 3 columns
v = [3, 1, 2] 3 numbers
W × v = [1×3 + 2×1 + 0×2, 0×3 + 1×1 + (−1)×2]
= [5, −1] 2 numbersThe rule for shapes: an (m × n) matrix needs a vector of n numbers and gives back m numbers. The inner sizes must match, or the multiplication isn't defined. Most shape errors in ML code are this rule being broken.
A layer's weight matrix is 512 × 768. How many numbers must the input vector have?
Where you'll see this
A layer that maps 4,096 numbers to 4,096 numbers is a 4,096 × 4,096 matrix plus a bias: 16,781,312 learned numbers. Large models are mostly stacks of matrices like this.
GPUs are built to multiply matrices very fast, which is why deep learning runs on them. When people say a model needs a lot of compute, they mostly mean a lot of matrix multiplication.
Common mistakes
- Mixing up shapes: an (m × n) matrix needs a vector of length n and returns one of length m.
- Assuming matrix multiplication is commutative. A·B and B·A are usually different, and may not both exist.
- Treating a matrix as just a table of numbers instead of a function that transforms vectors.
Answer in your own words, out loud or on paper, before you open the model answer.
What does it mean to multiply a matrix by a vector?
Multiplying a matrix by a vector applies a transformation to it. Each output entry is the dot product of one matrix row with the vector, so an m-by-n matrix turns a length-n vector into a length-m one. Geometrically, the columns tell you where each basis direction lands, and the output is the matching mix of those columns. That's how one matrix can rotate, stretch, project or change the number of dimensions. In neural networks, every linear layer is exactly this: the weight matrix maps the input features to a new set of features, and stacking layers composes those transformations, with non-linearities in between so the whole thing isn't just one big matrix.
- How many parameters does a fully connected layer from 1,024 to 512 inputs have?
- Why do neural networks run on GPUs?
Go further
- VideoEssence of linear algebra (series)3Blue1Brown
- Video18.06 Linear Algebra (lectures)Gilbert Strang, MIT OpenCourseWare