A big model is smart but slow and expensive. Distillation trains a small model to imitate it.
3 minute read
In short
Distillation trains a small student model to imitate a large teacher.
Soft probabilities carry more information than hard labels; temperature softens them further.
For LLMs, the student is often fine-tuned on filtered teacher outputs.
In plain words
Knowledge distillation uses a large teacher model to train a smaller student. The student learns not just the right answers but the teacher's full sense of which answers are plausible. The result is a model that is much cheaper to run and keeps most of the teacher's skill on the target task.
Worked example: soft labels
A teacher classifies a review into positive, neutral or negative, with scores (logits) 3.0, 1.0 and 0.2. A hard label would just say 'positive'. The teacher's probabilities carry more: neutral is a real possibility, negative isn't.
temperature
positive
neutral
negative
1
0.836
0.113
0.051
2
0.619
0.228
0.153
4
0.475
0.288
0.236
Raising the temperature softens the probabilities, making the teacher's 'second opinions' easier for the student to learn from. The student is trained to match these soft probabilities, usually alongside the true labels.
Distillation with LLMs
1
Collect prompts
Real or realistic inputs for the task, for example 20,000 support tickets.
2
Let the teacher answer
Run the big model to produce high-quality outputs, and filter out bad ones.
3
Fine-tune the student
Train a small model on the teacher's outputs.
4
Compare on a held-out test set
Check quality, cost and latency side by side before switching.
Your turn
Why train the student on the teacher's probabilities instead of only the correct label?
What to remember
Distillation is how many fast, cheap models are made. Check the terms of service before training on a provider's outputs; some forbid using them to build competing models. And remember the student can't exceed the teacher on the task, so pick a teacher that's actually good at it.
Common mistakes
Expecting the student to match the teacher on everything; it usually matches on the narrow task it was trained for.
Distilling from a model whose terms of use forbid it.
Skipping evaluation on hard cases, where small students lose the most quality.
Check yourself
Answer in your own words, out loud or on paper, before you open the model answer.
What is knowledge distillation and why use it?
Knowledge distillation trains a smaller student model to imitate a larger teacher. Instead of learning only from hard labels, the student can learn from the teacher's full probability distribution, which carries extra information, like which wrong answers are nearly right. With LLMs, a common version is simpler: use the teacher to generate high-quality answers for many prompts in your domain, then fine-tune the student on them. The point is cost and speed: a student might be ten times cheaper and faster while keeping most of the quality on that task. The trade-off is that students lose quality on unusual or hard inputs, so you evaluate carefully and often route those cases to the bigger model.
More to think about
Why use soft labels and a temperature in distillation?
How would you make a cheaper model for one task without losing much quality?