Lesson 41 · 2 min
Distillation: a big model teaches a small one
Distillation trains a small, fast model to copy the answers of a big one, so you keep much of the skill at a fraction of the cost.
Big models are clever but slow and expensive to run. Many everyday tasks do not need all that power, so companies want smaller models that behave almost as well.
In distillation, the big model plays teacher. It answers a huge set of questions, and the small model, the student, is trained to give the same answers. It can even learn from how sure the teacher was about each choice, which carries more information than a plain right or wrong.
The student ends up much smaller and quicker. It usually loses some skill, especially on hard or unusual questions, but for common tasks the gap can be small.
Many of the "mini", "lite" or "fast" versions of popular models are made partly this way. That is why a small model can feel surprisingly capable on simple jobs and then stumble on tricky ones.
Do it now
Test a small model against a big one
- 1. Pick a chatbot that offers a fast or mini model and a larger one.
- 2. Ask both the easy question and the hard question in the prompt below.
- 3. Notice where the small model keeps up and where it falls behind.
Answer two questions. 1) Rewrite this sentence to sound more polite: "Send me the report now." 2) A shop sells pens at 3 for 10,000 dong and notebooks at 2 for 15,000 dong. I buy 7 pens and 5 notebooks, and the shop only sells in full packs. What is the cheapest way to get at least that many, and what does it cost? Show your working.
Quick check
Words from this lesson
- Distillation
- Training a smaller, cheaper model to copy the answers of a bigger one, keeping much of its skill.
- Small language model
- A compact language model that is cheaper and faster, often good enough for focused tasks and able to run on devices.