epochs

Learn · Module 06 · Transformers · Lesson 4 of 6

The context window and the KV cache

A model can only read so much at once, and generating text would be painfully slow without one simple trick.

4 minute read

In short
  • The context window is the model's only memory: prompt and answer must fit inside it.
  • The KV cache saves each token's keys and values, so each new token costs one computation instead of a full re-run.
  • Long prompts slow the first token; shared prompt starts can be cached across requests.

This lesson is part of Pro

The first two lessons of every module are free. Pro opens every lesson and every project.

See Pro plansFrom ₹119 a month. No auto-renewal.