Learn · Module 12 · Inference and serving · Lesson 1 of 6
Latency and throughput
Users feel latency; your bill feels throughput. They pull in opposite directions, and you need words for both.
- Reply time ≈ time to first token + time per token × output tokens.
- TTFT depends on the prompt; TPOT on the model and hardware; throughput on batching.
- Shorter outputs and streaming are the cheapest speed-ups.
In plain words
Three numbers describe how fast a model responds. Time to first token (TTFT): how long before anything appears. Time per output token (TPOT): how fast text streams after that. Throughput: how many tokens per second the whole server produces across all users.
Total time for one reply is roughly TTFT plus TPOT times the number of output tokens. Serving many users at once (batching) raises throughput but can slow each individual reply.
Worked example: how long will this answer take?
- 1Measure the two parts
The prompt takes 0.4 seconds to process. Then each token takes 20 milliseconds.
TTFT = 0.4 s TPOT = 20 ms (50 tokens a second)
- 2A 200-token answer
0.4 + 200 × 0.020 = 4.4 seconds
- 3Where to cut
Output length dominates here: halving the answer to 100 tokens saves 2 seconds; halving TTFT saves only 0.2.
| goal | what usually helps |
|---|---|
| Lower TTFT | Shorter prompts, prompt caching, a closer region |
| Lower TPOT | Smaller or quantised model, faster hardware |
| Shorter total time | Ask for shorter answers; stream them |
| Higher throughput | Batch requests together |
Users complain a chat feels slow, but replies are only 30 tokens long. TTFT is 3 seconds. What should you look at first?
What to remember
Always stream replies in chat, since watching text appear feels far faster than waiting for all of it. Measure TTFT and TPOT separately; they have different causes and different fixes. Report percentiles, not averages, as the benchmarking lesson explains.
Common mistakes
- Quoting one latency number without saying whether it's time to first token or total time.
- Maximising batch size for throughput and hurting each user's response time.
- Measuring with short prompts when real traffic has long ones.
Answer in your own words, out loud or on paper, before you open the model answer.
What's the difference between latency and throughput for LLMs?
Latency is how long one request takes, from the user's point of view. For LLMs it has two parts: time to first token, which users feel as responsiveness, and the time to generate the rest, which depends on output length and tokens per second. Throughput is how much total work the system does, like requests or tokens per second across all users. They trade off: batching many requests together uses the GPU better, so throughput goes up, but each request may wait or generate more slowly, so latency goes up. A chat product cares most about time to first token and smooth streaming, while an overnight batch job cares about throughput and cost.
- How would you make an LLM feature feel faster?
- What is time to first token, and what affects it?
Go further
- ArticleHow continuous batching enables 23x throughput in LLM inferenceAnyscale
- ArticleLLM Inference Performance Engineering: Best PracticesDatabricks
- PaperOrca: A Distributed Serving System for Transformer ModelsYu et al., OSDI 2022