Learn · Module 12 · Inference and serving · Lesson 2 of 6
Where the time goes
Reading the prompt and writing the answer are two very different jobs for the hardware. Knowing which is the bottleneck tells you what will help.
- Prefill processes the prompt in parallel and is limited by compute.
- Decode makes one token at a time and is limited by memory bandwidth.
- Batching shares weight reads across users, multiplying throughput.
In plain words
Inference has two phases. Prefill processes the whole prompt at once; it's limited by raw computing power. Decode generates tokens one at a time, and for each one the GPU must read every weight of the model from memory; it's limited by memory bandwidth, how fast data moves from memory to the chips.
Worked example: the speed limit of decoding
- 1Size of a 7B model in bfloat16
7 × 10⁹ × 2 bytes = 14 GB
- 2One token means reading all of it
A modern GPU moves about 2 TB per second.
14 GB ÷ 2,000 GB/s = 7 ms per token
- 3So, at most
That's a hard ceiling for one user, no matter how fast the maths is.
1 ÷ 0.007 ≈ 143 tokens per second
- 4Batching changes everything
If 32 users' tokens are computed together, the weights are read once for all 32.
32 tokens per 7 ms ≈ 4,571 tokens per second
Worked example: prefill
Processing a token takes about 2 × parameters operations. A 1,000-token prompt on a 7B model is about 2 × 7×10⁹ × 1,000 = 1.4 × 10¹³ operations. At 300 × 10¹² operations per second of usable compute, that's about 47 milliseconds. Prompts are processed in parallel, which is why a long prompt is much cheaper per token than a long answer.
Why does batching many users together raise throughput so much during decoding?
What to remember
Prefill is compute-bound and parallel; decode is memory-bound and sequential. That's why output tokens cost several times more than input tokens with most providers, why quantisation (smaller weights to read) speeds up decoding, and why servers batch aggressively. The KV cache also lives in memory and competes with batch size.
Common mistakes
- Assuming a long prompt is free because the answer is short; prefill takes time too.
- Thinking a faster GPU always helps decode; decode is mostly limited by memory bandwidth.
- Ignoring that prompt caching mainly speeds up prefill.
Answer in your own words, out loud or on paper, before you open the model answer.
What are the prefill and decode phases?
Generating a response has two phases. Prefill processes the whole prompt at once: all input tokens go through the model in parallel, filling the KV cache. It's compute-heavy and determines the time to first token, so long prompts mean a longer wait before anything appears. Decode then generates one token at a time, each step reading the model weights and the KV cache from memory to produce a single token. That's limited by memory bandwidth rather than compute, so it's relatively slow per token. This is why long outputs take time, why batching helps decode efficiency, and why techniques like prompt caching help prefill while quantisation and speculative decoding help decode.
- Why are output tokens more expensive than input tokens?
- Estimate the maximum tokens per second for a 7B model on one GPU.
Go further
- ArticleTransformer Inference Arithmetickipply
- PaperFast Inference from Transformers via Speculative DecodingLeviathan et al., 2022