epochs

Learn · Module 04 · Deep learning in PyTorch · Lesson 2 of 3

A training loop

Every PyTorch model is trained by the same five lines. Learn them once and you can read almost any training code.

4 minute read

In short
  • Training is forward, loss, zero_grad, backward, step, repeated.
  • Use losses that take raw scores (logits), like BCEWithLogitsLoss and CrossEntropyLoss.
  • A DataLoader serves shuffled mini-batches; one pass over the data is an epoch.

In plain words

You build a model from layers, pick a loss function and an optimiser, then repeat: run the model, measure the loss, clear old gradients, backpropagate, and step. In the NumPy neural network project this took about 40 lines; in PyTorch it's about 15, because layers, gradients and Adam are built in.

Worked example: the circle problem in PyTorch

The same 400 points as the NumPy project: label 1 if a point is inside a circle of radius 0.6.

Model, loss, optimiser and the loop
from torch import nn

model = nn.Sequential(nn.Linear(2, 16), nn.ReLU(), nn.Linear(16, 1))
opt = torch.optim.Adam(model.parameters(), lr=0.05)
loss_fn = nn.BCEWithLogitsLoss()   # sigmoid + log loss, in one stable step

for step in range(300):
    logits = model(X).squeeze(1)
    loss = loss_fn(logits, Y)
    opt.zero_grad()     # 1. clear old gradients
    loss.backward()     # 2. backpropagate
    opt.step()          # 3. update the weights

# step 0: loss 0.721
# step 100: loss 0.099
# step 299: loss 0.020
# accuracy 1.0, with 65 parameters
  1. 1
    Count the parameters

    The first layer has 2 × 16 weights and 16 biases; the second has 16 weights and 1 bias.

    32 + 16 + 16 + 1 = 65
  2. 2
    Why BCEWithLogitsLoss?

    The model outputs raw scores (logits). This loss applies the sigmoid inside, in a way that can't overflow, so the model never needs its own sigmoid layer during training.

  3. 3
    Why Adam?

    It adapts the step size for each weight, so it works well with little tuning. From the Optimisers lesson.

Bigger data: mini-batches

With thousands of examples you don't feed them all at once. A DataLoader shuffles the data and serves it in batches, say 64 at a time, and the same three lines run for each batch. One pass over all the batches is an epoch. The digit recogniser project uses exactly this.

Your turn

A training loop forgets opt.zero_grad(). What's the most likely result?

What to remember

Model, loss function, optimiser; then forward, loss, zero_grad, backward, step. Output raw scores and use a loss that includes the softmax or sigmoid. Use a DataLoader for mini-batches once the data doesn't fit in one go.

Common mistakes

  • Applying sigmoid or softmax in the model and then using a loss that applies it again.
  • Forgetting zero_grad(), so gradients from earlier steps pile up.
  • Evaluating on the training data and calling it test accuracy.
Check yourself

Answer in your own words, out loud or on paper, before you open the model answer.

Walk me through a PyTorch training loop.

First I define the model, usually as layers in nn.Sequential or an nn.Module subclass, then a loss function and an optimiser like Adam over model.parameters(). A DataLoader serves shuffled mini-batches. For each batch: run the model to get raw scores, compute the loss against the labels, call optimizer.zero_grad() to clear old gradients, loss.backward() to compute new ones, and optimizer.step() to update the weights. One pass over all batches is an epoch. I'd print the training loss every few epochs, and after each epoch switch to eval mode and measure validation accuracy under torch.no_grad(), so I can spot overfitting and stop at the right time.

More to think about
  • Why output logits instead of probabilities when training?
  • What is an epoch, and what is a batch?

Go further

Was this lesson clear?