Learn · Module 06 · Transformers · Lesson 4 of 6
The context window and the KV cache
A model can only read so much at once, and generating text would be painfully slow without one simple trick.
4 minute read
- The context window is the model's only memory: prompt and answer must fit inside it.
- The KV cache saves each token's keys and values, so each new token costs one computation instead of a full re-run.
- Long prompts slow the first token; shared prompt starts can be cached across requests.
In plain words
The context window is the most tokens a model can handle in one go: your prompt plus its answer. Anything outside the window doesn't exist for the model. It has no other memory.
Generation happens one token at a time. To produce each new token, attention needs the keys and values of every earlier token. Recomputing them for every new token would repeat the same work again and again.
The KV cache fixes that. The keys and values of earlier tokens never change, so the model computes them once, saves them, and for each new token computes only that token's own.
Worked example: four new tokens after a 100-token prompt
- 1Without a cache
Each new token means running the model over everything so far.
101 + 102 + 103 + 104 = 410 token computations
- 2With a cache
Process the 100-token prompt once and save its keys and values. Then each new token is just one more.
100 + 1 + 1 + 1 + 1 = 104 token computations
- 3Compare
About four times less work here, and the gap widens with every extra token generated.
410 ÷ 104 = 3.9×
This is why the first token of a reply takes longest. The model must process the whole prompt before it can say anything. After that, each token is quick.
Worked example: what the cache costs in memory
Speed isn't free. The cache stores a key and a value for every token in every layer. For GPT-2 small (12 layers, 768 numbers per vector, 4 bytes per number) and our 104 tokens:
12 layers × 104 tokens × 2 (key and value) × 768 numbers × 4 bytes = 7,667,712 bytes ≈ 7.7 MB
That's small. But a large model with a 100,000-token context needs many gigabytes of cache for a single conversation. Memory for the cache, not raw compute, is often what limits how many users one GPU can serve.
A user sends a 50,000-token document and asks for a one-sentence summary. Where does most of the waiting time go?
What it means for your product
Model providers reuse this idea across requests. If many requests start with the same long text, such as a system prompt or a document, the provider can keep the cache for that shared start and skip recomputing it. This is called prompt caching, and cached tokens are usually billed at a lower price.
To benefit, put the part that never changes at the start of the prompt and the part that changes at the end. A cache only helps up to the first token that differs.
Common mistakes
- Forgetting that the cache grows with sequence length and batch size, and often limits how many users fit on a GPU.
- Thinking the cache speeds up the first token. It speeds up every token after it.
- Caching queries too. Only keys and values from earlier tokens are reused.
Answer in your own words, out loud or on paper, before you open the model answer.
What is the KV cache and why does it matter?
When a model generates text, each new token needs to attend to every earlier token, which needs their keys and values. Without a cache, you'd recompute keys and values for the whole sequence at every step, so total work grows with the square of the length. The KV cache stores each layer's keys and values for tokens already processed, so each new step only computes them for the one new token and reads the rest from memory. That makes generation dramatically faster. The cost is memory: it grows with layers, heads, sequence length and batch size, and for long contexts it can exceed the model weights, which is why techniques like grouped-query attention and paged caches exist.
- Why is time to first token different from time per token?
- How would you structure prompts to make the most of prompt caching?
Go further
- ArticleKV Caching ExplainedHugging Face community
- PaperEfficient Memory Management for LLM Serving with PagedAttention (vLLM)Kwon et al., 2023
- PaperGQA: Training Generalized Multi-Query Transformer ModelsAinslie et al., 2023
This lesson is part of Pro
The first two lessons of every module are free. Pro opens every lesson and every project.
See Pro plansFrom ₹119 a month. No auto-renewal.