Learn · Module 07 · Training and fine-tuning · Lesson 3 of 8
Training at scale
A 7-billion-parameter model doesn't fit on one GPU for training. A little arithmetic shows why, and what teams do about it.
In short
- Training with Adam needs about 16 bytes per parameter, roughly 8 times the weights alone.
- Training compute is about 6 × parameters × tokens operations.
- Split work with data parallelism, sharding, tensor or pipeline parallelism depending on what doesn't fit.
This lesson is part of Pro
The first two lessons of every module are free. Pro opens every lesson and every project.
See Pro plansFrom ₹119 a month. No auto-renewal.