L043 · Under the hood · 3 min
Mixture of experts: a model that only wakes up the parts it needs
A mixture-of-experts model is split into many smaller sub-networks, and for each piece of text it switches on only a few of them, which saves a lot of computing.
01 · The idea
A normal model uses all of its parameters for every word it writes. The bigger it gets, the more work each word costs.
A mixture-of-experts model divides much of that work among many "experts", which are smaller parts of the network. A small router looks at each token and sends it to the few experts that seem most useful. The rest stay idle for that step.
So the model can be very large in total while each word only uses a slice of it. That makes it cheaper and faster to run than a model of the same total size where everything works all the time.
The experts are not neat specialists like a lawyer and a doctor. They learn their own patterns during training, and those patterns are often hard for people to label. The whole model still needs lots of memory, because every expert has to be loaded and ready.
02 · Think of it like this
It is like a hospital reception desk. Each patient is sent to two or three of the right departments, not to every doctor in the building. The hospital is big, but each visit only uses a small part of it.
03 · Do it now
04 · Quick check
Words in this lesson
See also
L067Under the hood · 2 min
Small models and large models
Bigger models know more and reason better, while smaller ones are faster, cheaper and can run on your own device.
L002How AI thinks · 2 min
How machines learn
A machine learns by guessing, checking how wrong it was, and adjusting itself a tiny bit, billions of times.
L041Under the hood · 2 min
Distillation: a big model teaches a small one
Distillation trains a small, fast model to copy the answers of a big one, so you keep much of the skill at a fraction of the cost.
L045Under the hood · 2 min
Quantization: shrinking a model to fit
Quantization stores a model's numbers with less precision, so it takes up much less memory and can run on smaller computers, with a small drop in quality.