epochs

Learn · Module 12 · Inference and serving · Lesson 4 of 6

Caching

The fastest, cheapest model call is the one you don't make again. Caching works at several levels.

3 minute read

In short
  • Caches work at several levels: KV, prompt prefix, exact response and semantic.
  • Prompt caching reuses a shared prompt start and can cut input cost sharply.
  • Put stable content first; semantic caches need care to avoid wrong answers.

This lesson is part of Pro

The first two lessons of every module are free. Pro opens every lesson and every project.

See Pro plansFrom ₹119 a month. No auto-renewal.