ManuAIVI

L044 · Under the hood · 2 min

Training vs inference: learning once, answering millions of times

Training is when a model learns, and it happens before you ever use it; inference is the model answering you, using what it already learned.

01 · The idea

Training is the long, expensive part. A company runs huge amounts of data through thousands of chips for weeks or months, slowly adjusting the model's weights until it gets good at predicting text. When it finishes, the weights are frozen.

Inference is everything after that. Each time you send a message, the frozen model reads it and works out a reply. One answer costs very little, but millions of people asking all day adds up, so serving a popular model also uses a lot of computing.

This explains a common surprise. The model does not learn from your chat as you talk. If it seems to remember you, that comes from memory features or saved notes added on top, which the app feeds back in as text.

Your conversations might be used to train a future version, depending on the service and your settings. That is a separate, later step, and you can often switch it off in the privacy settings.

02 · Think of it like this

Training is a student spending years at medical school. Inference is the doctor seeing patients. Each appointment uses what was learned; the doctor does not go back to school between patients.

03 · Do it now

04 · Quick check

You correct a chatbot's mistake in a chat. What happens to the model itself?

Words in this lesson

See also