epochs

Learn · Module 03 · Neural networks · Lesson 4 of 7

Optimisers

Gradient descent says which way to step. Optimisers decide how big each step should be, and they make a large difference.

4 minute read

In short
  • SGD steps by learning rate × gradient and is easily thrown by noise.
  • Momentum averages past gradients, smoothing noise and speeding up on consistent slopes.
  • Adam scales each weight's step by its gradient history; AdamW is the transformer default.

This lesson is part of Pro

The first two lessons of every module are free. Pro opens every lesson and every project.

See Pro plansFrom ₹119 a month. No auto-renewal.