epochs

Learn · Module 12 · Inference and serving · Lesson 6 of 6

Benchmarking honestly

Averages hide the slow requests that users actually remember. Measure the whole spread.

3 minute read

In short
  • Use percentiles: p50 for typical, p95 and p99 for the slow tail.
  • Averages are pulled up by a few slow requests and hide what users experience.
  • Compare fairly: same prompts and output lengths, warmed up, at realistic load.

This lesson is part of Pro

The first two lessons of every module are free. Pro opens every lesson and every project.

See Pro plansFrom ₹119 a month. No auto-renewal.