Training is half the job. Running the model, saving it and moving it to a GPU have a few rules that trip people up.
3 minute read
In short
Use model.eval() and torch.no_grad() when predicting.
Save and load the state dict into the same model class.
Move the model and every batch to the same device.
In plain words
When you use a model to predict, three things change. Switch it to evaluation mode with model.eval(), so layers like dropout behave for prediction. Wrap the call in torch.no_grad(), so PyTorch doesn't store anything for backpropagation, which saves memory and time. And turn raw scores into probabilities yourself with sigmoid or softmax.
Predicting with the circle model
model.eval()
with torch.no_grad():
scores = model(torch.tensor([[0.0, 0.0], [0.9, 0.9]]))
probs = torch.sigmoid(scores).squeeze(1)
print(probs) # [1.0, 0.0]: the centre is inside, the corner outside
Saving and loading
Save the state dict, the dictionary of learned weights, not the whole Python object. To load, build the same model class and pour the weights in.
Save, then load into a fresh model
torch.save(model.state_dict(), "circle.pt")
fresh = nn.Sequential(nn.Linear(2, 16), nn.ReLU(), nn.Linear(16, 1))
fresh.load_state_dict(torch.load("circle.pt"))
fresh.eval()
# fresh(X) gives exactly the same outputs as model(X): True
GPUs
Pick a device once, then move the model and every batch to it. Tensors on different devices can't be mixed, which causes the most common PyTorch error message.
Device handling
device = "cuda" if torch.cuda.is_available() else "cpu"
model = model.to(device)
for xb, yb in loader:
xb, yb = xb.to(device), yb.to(device)
...
Your turn
Your model's test accuracy changes every time you run it on the same data. What did you most likely forget?
What to remember
For prediction: eval mode, no_grad, then sigmoid or softmax. Save state dicts and load them into the same architecture. Keep the model and its data on the same device.
Common mistakes
Forgetting model.eval(), so dropout and batch norm behave as in training.
Saving the whole model object with pickle, which breaks when the code changes; save the state dict instead.
Moving the model to the GPU but not the input tensors.
Check yourself
Answer in your own words, out loud or on paper, before you open the model answer.
What's the difference between model.eval() and torch.no_grad()?
They do different jobs. model.eval() changes how certain layers behave: dropout stops dropping activations and batch norm uses its stored averages instead of the current batch's statistics, so predictions become deterministic. torch.no_grad() is about memory and speed: it tells autograd not to record operations, because you won't call backward, so no intermediate values are kept. For inference you want both: eval mode for correct outputs and no_grad for efficiency. Forgetting eval gives noisy, slightly wrong results; forgetting no_grad gives correct results but wastes memory, which matters on a GPU serving many requests.
More to think about
How do you save and load a PyTorch model?
What causes 'expected all tensors to be on the same device' errors?