epochs

Learn · Module 07 · Training and fine-tuning · Lesson 2 of 8

Scaling laws

Bigger models get better in a surprisingly predictable way. That predictability decides how labs spend millions on compute.

3 minute read

In short
  • Loss falls predictably as parameters, data and compute grow, following power laws.
  • For a fixed budget, about 20 tokens per parameter minimises loss (Chinchilla).
  • Teams often train smaller models on more data to make running them cheaper.

In plain words

Scaling laws are measured relationships between a model's loss and three things: the number of parameters, the number of training tokens, and the compute spent. Loss falls smoothly as each grows, following a power law, so teams can train small models, fit the curve, and predict how a much bigger run will do before paying for it.

Model size versus data

For a fixed compute budget you can train a bigger model on fewer tokens, or a smaller one on more. DeepMind's Chinchilla study (Hoffmann et al., 2022) found the best balance is roughly 20 training tokens per parameter, and that many earlier models were too big for the data they saw.

Worked example: spending a budget

  1. 1
    Budget: 10²³ operations

    Compute ≈ 6 × N × D, with D ≈ 20 × N, so compute ≈ 120 × N².

    N = √(10²³ ÷ 120) ≈ 29 billion parameters
  2. 2
    Tokens for that size
    D = 20 × 29 billion ≈ 577 billion tokens
  3. 3
    Compare GPT-3

    175 billion parameters on 300 billion tokens used about 3.15 × 10²³ operations.

    same budget, Chinchilla-style: ≈ 51B parameters on ≈ 1 trillion tokens

In practice many teams now train beyond the 20-to-1 ratio on purpose. A smaller model trained on far more data costs more to train but is cheaper and faster to run for millions of users, so the total cost is lower.

Your turn

You have a fixed training budget. According to Chinchilla-style scaling, what's usually the better choice?

What to remember

Scaling laws predict loss, not specific skills; some abilities appear suddenly or depend heavily on data quality. Good data can be worth more than more data, and the supply of high-quality text is limited, which is why teams invest in filtering, synthetic data and new sources.

Common mistakes

  • Saying scaling laws predict specific abilities; they predict loss.
  • Assuming bigger is always better for a fixed budget, ignoring the data side.
  • Ignoring inference cost when choosing model size.
Check yourself

Answer in your own words, out loud or on paper, before you open the model answer.

What do scaling laws tell us?

Scaling laws are empirical relationships showing that a language model's loss falls smoothly and predictably, as a power law, as you increase parameters, training data and compute. That lets teams run small experiments, fit the curve and forecast a large run before paying for it. The Chinchilla work showed that for a fixed compute budget, the best loss comes from balancing model size and data, roughly 20 tokens per parameter, and that many earlier models were undertrained. In practice teams often train smaller models on even more data, because a smaller model is cheaper to serve. The limits are that loss doesn't map neatly to specific skills, and data quality matters a lot.

More to think about
  • Given a compute budget, how would you choose model size and data size?
  • Why might you train a model on far more data than the compute-optimal amount?

Go further

Was this lesson clear?