Learn · Module 07 · Training and fine-tuning · Lesson 2 of 8
Scaling laws
Bigger models get better in a surprisingly predictable way. That predictability decides how labs spend millions on compute.
- Loss falls predictably as parameters, data and compute grow, following power laws.
- For a fixed budget, about 20 tokens per parameter minimises loss (Chinchilla).
- Teams often train smaller models on more data to make running them cheaper.
In plain words
Scaling laws are measured relationships between a model's loss and three things: the number of parameters, the number of training tokens, and the compute spent. Loss falls smoothly as each grows, following a power law, so teams can train small models, fit the curve, and predict how a much bigger run will do before paying for it.
Model size versus data
For a fixed compute budget you can train a bigger model on fewer tokens, or a smaller one on more. DeepMind's Chinchilla study (Hoffmann et al., 2022) found the best balance is roughly 20 training tokens per parameter, and that many earlier models were too big for the data they saw.
Worked example: spending a budget
- 1Budget: 10²³ operations
Compute ≈ 6 × N × D, with D ≈ 20 × N, so compute ≈ 120 × N².
N = √(10²³ ÷ 120) ≈ 29 billion parameters
- 2Tokens for that size
D = 20 × 29 billion ≈ 577 billion tokens
- 3Compare GPT-3
175 billion parameters on 300 billion tokens used about 3.15 × 10²³ operations.
same budget, Chinchilla-style: ≈ 51B parameters on ≈ 1 trillion tokens
In practice many teams now train beyond the 20-to-1 ratio on purpose. A smaller model trained on far more data costs more to train but is cheaper and faster to run for millions of users, so the total cost is lower.
You have a fixed training budget. According to Chinchilla-style scaling, what's usually the better choice?
What to remember
Scaling laws predict loss, not specific skills; some abilities appear suddenly or depend heavily on data quality. Good data can be worth more than more data, and the supply of high-quality text is limited, which is why teams invest in filtering, synthetic data and new sources.
Common mistakes
- Saying scaling laws predict specific abilities; they predict loss.
- Assuming bigger is always better for a fixed budget, ignoring the data side.
- Ignoring inference cost when choosing model size.
Answer in your own words, out loud or on paper, before you open the model answer.
What do scaling laws tell us?
Scaling laws are empirical relationships showing that a language model's loss falls smoothly and predictably, as a power law, as you increase parameters, training data and compute. That lets teams run small experiments, fit the curve and forecast a large run before paying for it. The Chinchilla work showed that for a fixed compute budget, the best loss comes from balancing model size and data, roughly 20 tokens per parameter, and that many earlier models were undertrained. In practice teams often train smaller models on even more data, because a smaller model is cheaper to serve. The limits are that loss doesn't map neatly to specific skills, and data quality matters a lot.
- Given a compute budget, how would you choose model size and data size?
- Why might you train a model on far more data than the compute-optimal amount?
Go further
- PaperScaling Laws for Neural Language ModelsKaplan et al., 2020
- PaperTraining Compute-Optimal Large Language Models (Chinchilla)Hoffmann et al., 2022