Lesson 45 · 2 min
Quantization: shrinking a model to fit
Quantization stores a model's numbers with less precision, so it takes up much less memory and can run on smaller computers, with a small drop in quality.
A model is mostly a giant list of numbers called weights. Normally each one is stored with lots of decimal detail, and with billions of them, the file is too big for an ordinary laptop or phone.
Quantization rounds those numbers to a coarser scale. Instead of keeping every tiny decimal, each weight is stored in fewer bits. The model can shrink to a half, a quarter or even less of its size.
Smaller means it fits in less memory and often runs faster. The cost is a little accuracy. With moderate rounding, most people cannot tell the difference. Squeeze too hard and answers get noticeably sloppier.
If you ever download a model to run yourself, you will see several sizes of the same model. Those are usually different quantization levels: pick the largest one your machine can comfortably hold.
Do it now
See rounding at work
- 1. Write down a price like 47,836 dong.
- 2. Round it to the nearest thousand, then to the nearest ten thousand.
- 3. Ask yourself when the rounded number stops being useful. A quantized model faces the same trade-off, billions of times over.
Quick check
Words from this lesson
- Quantization
- Storing a model's numbers with less precision so it uses less memory and runs faster, with a small loss of quality.
- Weights
- The learned numbers that make up a trained model. Sharing a model usually means sharing its weights file.
- Local model
- An AI model that runs on your own computer or phone, so your data does not leave the device.