epochs

Learn · Module 04 · Deep learning in PyTorch · Lesson 1 of 3

Tensors and autograd

PyTorch is NumPy that can run on a GPU and work out gradients for you. Those two ideas are most of what you need.

4 minute read

In short
  • A tensor is an array that can run on a GPU and remember how it was computed.
  • loss.backward() fills in .grad for every tensor with requires_grad=True.
  • Gradients accumulate, so clear them before each training step.

In plain words

A tensor is PyTorch's array: numbers with a shape, like a NumPy array. The difference is that PyTorch can remember how each tensor was computed. Mark a tensor with requires_grad=True, do some maths, call .backward() on the result, and PyTorch fills in the gradient of every marked tensor. That's autograd, and it's the backpropagation from the Neural networks module, done automatically.

Worked example: one weight, one gradient

A tiny model predicts y = w × x + b. With x = 2, w = 3, b = 1 and a target of 10, how should w and b change? First by hand, then by PyTorch.

  1. 1
    Prediction
    y = 3 × 2 + 1 = 7
  2. 2
    Loss

    Squared error against the target.

    (7 − 10)² = 9
  3. 3
    Gradient for w, by hand

    Chain rule: d(loss)/dw = 2 × (y − target) × x.

    2 × (−3) × 2 = −12
  4. 4
    Gradient for b, by hand

    d(loss)/db = 2 × (y − target).

    2 × (−3) = −6
The same thing in PyTorch
import torch

x = torch.tensor(2.0)
t = torch.tensor(10.0)
w = torch.tensor(3.0, requires_grad=True)
b = torch.tensor(1.0, requires_grad=True)

y = w * x + b
loss = (y - t) ** 2
loss.backward()          # autograd works out every gradient

print(y.item(), loss.item(), w.grad.item(), b.grad.item())
# 7.0 9.0 -12.0 -6.0

Exactly the hand calculation. Both gradients are negative, so increasing w and b would reduce the loss, which makes sense: the prediction 7 is below the target 10.

Shapes work like NumPy

Matrix maths
a = torch.tensor([[1.0, 2.0], [3.0, 4.0]])
print(a.shape)          # torch.Size([2, 2])
print(a @ a)            # [[7, 10], [15, 22]]  matrix multiply
print(a.mean(dim=0))    # [2, 3]  average of each column
Your turn

You call loss.backward() twice in a row without resetting. What happens to w.grad?

What to remember

Tensors are arrays that remember how they were made. backward() fills in .grad for every tensor marked requires_grad. Gradients add up until you clear them, and .item() turns a one-number tensor into a plain Python number.

Common mistakes

  • Forgetting to clear gradients, so they add up across steps.
  • Calling .numpy() on a tensor that requires grad or lives on the GPU; detach and move it to the CPU first.
  • Using integer tensors where floats are needed; gradients only flow through floating-point tensors.
Check yourself

Answer in your own words, out loud or on paper, before you open the model answer.

What does autograd do in PyTorch?

Autograd is PyTorch's automatic differentiation. When you create tensors with requires_grad=True and compute with them, PyTorch records each operation in a graph. Calling backward() on a scalar, usually the loss, walks that graph in reverse using the chain rule and stores the gradient of the loss with respect to each of those tensors in its .grad attribute. For example, with y = w times x plus b and a squared-error loss, it gives the same gradients you'd derive by hand. That's why you never write backpropagation yourself in PyTorch. Gradients accumulate, so training loops clear them every step, and torch.no_grad() switches recording off when you only need predictions.

More to think about
  • Why do training loops call zero_grad()?
  • What's the difference between a PyTorch tensor and a NumPy array?

Go further

Was this lesson clear?