L041 · Under the hood · 2 min
Distillation: a big model teaches a small one
Distillation trains a small, fast model to copy the answers of a big one, so you keep much of the skill at a fraction of the cost.
01 · The idea
Big models are clever but slow and expensive to run. Many everyday tasks do not need all that power, so companies want smaller models that behave almost as well.
In distillation, the big model plays teacher. It answers a huge set of questions, and the small model, the student, is trained to give the same answers. It can even learn from how sure the teacher was about each choice, which carries more information than a plain right or wrong.
The student ends up much smaller and quicker. It usually loses some skill, especially on hard or unusual questions, but for common tasks the gap can be small.
Many of the "mini", "lite" or "fast" versions of popular models are made partly this way. That is why a small model can feel surprisingly capable on simple jobs and then stumble on tricky ones.
02 · Think of it like this
It is like a master chef training a cook for a busy street stall. The cook cannot do the whole fine-dining menu, but learns to make the ten most popular dishes almost as well, and much faster.
03 · Do it now
04 · Quick check
Words in this lesson
See also
L067Under the hood · 2 min
Small models and large models
Bigger models know more and reason better, while smaller ones are faster, cheaper and can run on your own device.
L168Under the hood · 2 min
Where AI apps live: cloud or your device
Some AI runs in big data centres far away and some runs on your own phone or laptop, and that choice affects speed, privacy, cost and quality.
L039Under the hood · 3 min
How fine-tuning works
Fine-tuning takes a model that already knows language and trains it a bit more on your own examples, so it picks up a style or a task.
L043Under the hood · 3 min
Mixture of experts: a model that only wakes up the parts it needs
A mixture-of-experts model is split into many smaller sub-networks, and for each piece of text it switches on only a few of them, which saves a lot of computing.