Lesson 44 · 2 min
Training vs inference: learning once, answering millions of times
Training is when a model learns, and it happens before you ever use it; inference is the model answering you, using what it already learned.
Training is the long, expensive part. A company runs huge amounts of data through thousands of chips for weeks or months, slowly adjusting the model's weights until it gets good at predicting text. When it finishes, the weights are frozen.
Inference is everything after that. Each time you send a message, the frozen model reads it and works out a reply. One answer costs very little, but millions of people asking all day adds up, so serving a popular model also uses a lot of computing.
This explains a common surprise. The model does not learn from your chat as you talk. If it seems to remember you, that comes from memory features or saved notes added on top, which the app feeds back in as text.
Your conversations might be used to train a future version, depending on the service and your settings. That is a separate, later step, and you can often switch it off in the privacy settings.
Do it now
Check what your chatbot keeps
- 1. Open the settings of the chatbot you use most.
- 2. Look for options about memory and about using your chats to improve the model.
- 3. Decide which ones you want on, and switch the others off.
Quick check
Words from this lesson
- Inference
- The moment a trained model is actually used to produce an answer, as opposed to the earlier learning phase.
- Training
- The process of feeding a model many examples and adjusting its internal numbers until its outputs improve.
- GPU
- A chip that does huge numbers of small calculations at once. First built for video games, now the main engine for AI.