epochs

Learn · Module 07 · Training and fine-tuning · Lesson 8 of 8

Distillation

A big model is smart but slow and expensive. Distillation trains a small model to imitate it.

3 minute read

In short
  • Distillation trains a small student model to imitate a large teacher.
  • Soft probabilities carry more information than hard labels; temperature softens them further.
  • For LLMs, the student is often fine-tuned on filtered teacher outputs.

This lesson is part of Pro

The first two lessons of every module are free. Pro opens every lesson and every project.

See Pro plansFrom ₹119 a month. No auto-renewal.