epochs

Learn · Module 12 · Inference and serving · Lesson 2 of 6

Where the time goes

Reading the prompt and writing the answer are two very different jobs for the hardware. Knowing which is the bottleneck tells you what will help.

3 minute read

In short
  • Prefill processes the prompt in parallel and is limited by compute.
  • Decode makes one token at a time and is limited by memory bandwidth.
  • Batching shares weight reads across users, multiplying throughput.

In plain words

Inference has two phases. Prefill processes the whole prompt at once; it's limited by raw computing power. Decode generates tokens one at a time, and for each one the GPU must read every weight of the model from memory; it's limited by memory bandwidth, how fast data moves from memory to the chips.

Worked example: the speed limit of decoding

  1. 1
    Size of a 7B model in bfloat16
    7 × 10⁹ × 2 bytes = 14 GB
  2. 2
    One token means reading all of it

    A modern GPU moves about 2 TB per second.

    14 GB ÷ 2,000 GB/s = 7 ms per token
  3. 3
    So, at most

    That's a hard ceiling for one user, no matter how fast the maths is.

    1 ÷ 0.007 ≈ 143 tokens per second
  4. 4
    Batching changes everything

    If 32 users' tokens are computed together, the weights are read once for all 32.

    32 tokens per 7 ms ≈ 4,571 tokens per second

Worked example: prefill

Processing a token takes about 2 × parameters operations. A 1,000-token prompt on a 7B model is about 2 × 7×10⁹ × 1,000 = 1.4 × 10¹³ operations. At 300 × 10¹² operations per second of usable compute, that's about 47 milliseconds. Prompts are processed in parallel, which is why a long prompt is much cheaper per token than a long answer.

Your turn

Why does batching many users together raise throughput so much during decoding?

What to remember

Prefill is compute-bound and parallel; decode is memory-bound and sequential. That's why output tokens cost several times more than input tokens with most providers, why quantisation (smaller weights to read) speeds up decoding, and why servers batch aggressively. The KV cache also lives in memory and competes with batch size.

Common mistakes

  • Assuming a long prompt is free because the answer is short; prefill takes time too.
  • Thinking a faster GPU always helps decode; decode is mostly limited by memory bandwidth.
  • Ignoring that prompt caching mainly speeds up prefill.
Check yourself

Answer in your own words, out loud or on paper, before you open the model answer.

What are the prefill and decode phases?

Generating a response has two phases. Prefill processes the whole prompt at once: all input tokens go through the model in parallel, filling the KV cache. It's compute-heavy and determines the time to first token, so long prompts mean a longer wait before anything appears. Decode then generates one token at a time, each step reading the model weights and the KV cache from memory to produce a single token. That's limited by memory bandwidth rather than compute, so it's relatively slow per token. This is why long outputs take time, why batching helps decode efficiency, and why techniques like prompt caching help prefill while quantisation and speculative decoding help decode.

More to think about
  • Why are output tokens more expensive than input tokens?
  • Estimate the maximum tokens per second for a 7B model on one GPU.

Go further

Was this lesson clear?