epochs

Learn · Module 07 · Training and fine-tuning · Lesson 3 of 8

Training at scale

A 7-billion-parameter model doesn't fit on one GPU for training. A little arithmetic shows why, and what teams do about it.

4 minute read

In short
  • Training with Adam needs about 16 bytes per parameter, roughly 8 times the weights alone.
  • Training compute is about 6 × parameters × tokens operations.
  • Split work with data parallelism, sharding, tensor or pipeline parallelism depending on what doesn't fit.

This lesson is part of Pro

The first two lessons of every module are free. Pro opens every lesson and every project.

See Pro plansFrom ₹119 a month. No auto-renewal.