epochs

Learn · Module 12 · Inference and serving · Lesson 3 of 6

Quantisation

Store each weight in fewer bits, and the model gets smaller and faster. The trick is losing as little quality as possible.

4 minute read

In short
  • Quantisation stores weights in fewer bits by rounding to a small set of levels.
  • A 7B model shrinks from 14 GB in 16-bit to 3.5 GB in 4-bit, and decodes faster.
  • Fewer bits mean bigger rounding errors; outliers make it worse. Measure the quality loss.

This lesson is part of Pro

The first two lessons of every module are free. Pro opens every lesson and every project.

See Pro plansFrom ₹119 a month. No auto-renewal.