Learn · Module 03 · Neural networks · Lesson 5 of 7
Keeping training stable
Deep networks multiply many numbers together. Multiply enough small numbers and you get nothing; enough big ones and you get infinity.
In short
- Gradients multiply through layers, so they can vanish to zero or explode to infinity.
- Initialisation, normalisation and residual connections keep signals at a steady scale.
- Gradient clipping and a lower learning rate stop single huge updates.
This lesson is part of Pro
The first two lessons of every module are free. Pro opens every lesson and every project.
See Pro plansFrom ₹119 a month. No auto-renewal.