Learn · Module 04 · Deep learning in PyTorch · Lesson 1 of 3
Tensors and autograd
PyTorch is NumPy that can run on a GPU and work out gradients for you. Those two ideas are most of what you need.
- A tensor is an array that can run on a GPU and remember how it was computed.
- loss.backward() fills in .grad for every tensor with requires_grad=True.
- Gradients accumulate, so clear them before each training step.
In plain words
A tensor is PyTorch's array: numbers with a shape, like a NumPy array. The difference is that PyTorch can remember how each tensor was computed. Mark a tensor with requires_grad=True, do some maths, call .backward() on the result, and PyTorch fills in the gradient of every marked tensor. That's autograd, and it's the backpropagation from the Neural networks module, done automatically.
Worked example: one weight, one gradient
A tiny model predicts y = w × x + b. With x = 2, w = 3, b = 1 and a target of 10, how should w and b change? First by hand, then by PyTorch.
- 1Prediction
y = 3 × 2 + 1 = 7
- 2Loss
Squared error against the target.
(7 − 10)² = 9
- 3Gradient for w, by hand
Chain rule: d(loss)/dw = 2 × (y − target) × x.
2 × (−3) × 2 = −12
- 4Gradient for b, by hand
d(loss)/db = 2 × (y − target).
2 × (−3) = −6
import torch x = torch.tensor(2.0) t = torch.tensor(10.0) w = torch.tensor(3.0, requires_grad=True) b = torch.tensor(1.0, requires_grad=True) y = w * x + b loss = (y - t) ** 2 loss.backward() # autograd works out every gradient print(y.item(), loss.item(), w.grad.item(), b.grad.item()) # 7.0 9.0 -12.0 -6.0
Exactly the hand calculation. Both gradients are negative, so increasing w and b would reduce the loss, which makes sense: the prediction 7 is below the target 10.
Shapes work like NumPy
a = torch.tensor([[1.0, 2.0], [3.0, 4.0]]) print(a.shape) # torch.Size([2, 2]) print(a @ a) # [[7, 10], [15, 22]] matrix multiply print(a.mean(dim=0)) # [2, 3] average of each column
You call loss.backward() twice in a row without resetting. What happens to w.grad?
What to remember
Tensors are arrays that remember how they were made. backward() fills in .grad for every tensor marked requires_grad. Gradients add up until you clear them, and .item() turns a one-number tensor into a plain Python number.
Common mistakes
- Forgetting to clear gradients, so they add up across steps.
- Calling .numpy() on a tensor that requires grad or lives on the GPU; detach and move it to the CPU first.
- Using integer tensors where floats are needed; gradients only flow through floating-point tensors.
Answer in your own words, out loud or on paper, before you open the model answer.
What does autograd do in PyTorch?
Autograd is PyTorch's automatic differentiation. When you create tensors with requires_grad=True and compute with them, PyTorch records each operation in a graph. Calling backward() on a scalar, usually the loss, walks that graph in reverse using the chain rule and stores the gradient of the loss with respect to each of those tensors in its .grad attribute. For example, with y = w times x plus b and a squared-error loss, it gives the same gradients you'd derive by hand. That's why you never write backpropagation yourself in PyTorch. Gradients accumulate, so training loops clear them every step, and torch.no_grad() switches recording off when you only need predictions.
- Why do training loops call zero_grad()?
- What's the difference between a PyTorch tensor and a NumPy array?
Go further
- DocsTensorsPyTorch tutorials
- DocsAutomatic differentiation with torch.autogradPyTorch tutorials