Lesson 43 · 3 min
Mixture of experts: a model that only wakes up the parts it needs
A mixture-of-experts model is split into many smaller sub-networks, and for each piece of text it switches on only a few of them, which saves a lot of computing.
A normal model uses all of its parameters for every word it writes. The bigger it gets, the more work each word costs.
A mixture-of-experts model divides much of that work among many "experts", which are smaller parts of the network. A small router looks at each token and sends it to the few experts that seem most useful. The rest stay idle for that step.
So the model can be very large in total while each word only uses a slice of it. That makes it cheaper and faster to run than a model of the same total size where everything works all the time.
The experts are not neat specialists like a lawyer and a doctor. They learn their own patterns during training, and those patterns are often hard for people to label. The whole model still needs lots of memory, because every expert has to be loaded and ready.
Do it now
Ask a model to explain its own design
- 1. Use the prompt below with any chatbot.
- 2. Check whether it explains the router and the experts in plain words.
- 3. Ask a follow-up: why does the whole model still need a lot of memory?
Explain a mixture-of-experts AI model to a 14-year-old using a school as the example. Cover what the experts are, what the router does, and why it saves effort. Under 150 words.
Quick check
Words from this lesson
- Mixture of experts
- A model design where only some parts of the network switch on for each request, making large models faster and cheaper.
- Parameters
- The adjustable numbers inside a model that are set during training. Big models have billions of them.