epochs

Learn · Module 07 · Training and fine-tuning · Lesson 1 of 8

Pretraining

Before a model can follow instructions, it reads a huge amount of text and learns one skill: predicting the next token.

4 minute read

In short
  • Pretraining teaches next-token prediction on trillions of tokens, producing a base model.
  • The loss is cross-entropy: −ln of the probability given to the right token.
  • Compute is about 6 × parameters × tokens, which is why few teams pretrain.

In plain words

Pretraining is the first and most expensive stage. The model reads trillions of tokens from the web, books and code. At every position it guesses the next token, and is corrected. Grammar, facts, reasoning patterns and coding style all come out of this one simple task.

The result is a base model. It's knowledgeable but not helpful yet: ask it a question and it might continue with three more questions, because that's what text on the web often does.

Worked example: the loss

For each position, the loss is −ln(p), where p is the probability the model gave to the token that actually came next. This is cross-entropy.

probability given to the right token0.90.50.10.01
loss0.1050.6932.3034.605
  1. 1
    Three tokens, three probabilities

    The model gave the right next tokens probabilities 0.6, 0.3 and 0.9.

    losses: 0.511, 1.204, 0.105
  2. 2
    Average them

    Training reports the average over billions of positions.

    (0.511 + 1.204 + 0.105) ÷ 3 = 0.607
  3. 3
    Perplexity

    e raised to the average loss. Roughly, the model is as unsure as if choosing between 1.8 equally likely tokens.

    e^0.607 = 1.83

Worked example: what it costs

A standard estimate: training takes about 6 × parameters × tokens floating-point operations. Research on scaling found that, for a fixed budget, about 20 training tokens per parameter works well.

  1. 1
    A 1-billion-parameter model on 20 billion tokens
    6 × 1×10⁹ × 20×10⁹ = 1.2 × 10²⁰ operations
  2. 2
    On one GPU

    At 10¹⁵ operations per second and 40% real-world efficiency:

    1.2×10²⁰ ÷ (0.4 × 10¹⁵) ÷ 3600 ≈ 83 hours
  3. 3
    Scale up

    A 7-billion-parameter model would want about 140 billion tokens, and frontier models use far more, so pretraining runs on thousands of GPUs for weeks.

Your turn

A model gives the correct next token a probability of 0.5. What's its loss on that token?

Where you'll see this

Few teams pretrain models; it's done by a handful of labs. What you'll do is choose between pretrained models and adapt them. Knowing that a base model is a next-token predictor explains a lot: why it continues text instead of answering, why its knowledge stops at a date, and why later stages are needed to make it useful.

Common mistakes

  • Saying pretraining teaches the model to follow instructions. That comes later, in fine-tuning.
  • Assuming more data always helps regardless of quality; deduplication and filtering matter a lot.
  • Ignoring that the model's knowledge stops at the data cut-off date.
Check yourself

Answer in your own words, out loud or on paper, before you open the model answer.

What does a model learn during pretraining?

Pretraining trains the model to predict the next token on a huge amount of text. That one simple objective forces it to learn a lot: grammar and style, facts that appear often, how arguments and code are structured, and some reasoning patterns, because all of these help predict what comes next. The result is a base model that's very good at continuing text but not at following instructions or being helpful; that's added by fine-tuning and preference training afterwards. Pretraining is by far the most expensive stage, and data quality matters enormously, so teams spend a lot of effort on filtering, deduplication and mixing different sources.

More to think about
  • What is perplexity?
  • Why can't a base model follow instructions well?

Go further

Was this lesson clear?