epochs

Learn · Module 03 · Neural networks · Lesson 5 of 7

Keeping training stable

Deep networks multiply many numbers together. Multiply enough small numbers and you get nothing; enough big ones and you get infinity.

3 minute read

In short
  • Gradients multiply through layers, so they can vanish to zero or explode to infinity.
  • Initialisation, normalisation and residual connections keep signals at a steady scale.
  • Gradient clipping and a lower learning rate stop single huge updates.

This lesson is part of Pro

The first two lessons of every module are free. Pro opens every lesson and every project.

See Pro plansFrom ₹119 a month. No auto-renewal.