How LLMs are trained

A large language model starts as a transformer with random weights. It reads a vast amount of text, trillions of tokens from the web, books and code, and learns one task: predict the next token. Every wrong guess produces a loss, and gradient descent nudges billions of weights.

To predict well, the model has to absorb grammar, facts, styles and some reasoning patterns. The result is a base model. It continues text fluently, but it does not yet follow instructions reliably. Pretraining a frontier model costs many millions of dollars in compute.

Next comes supervised fine-tuning. The base model trains further on a much smaller set of example conversations: a question and a good answer written or checked by people. This teaches the format and manners of an assistant.

You can fine-tune open models yourself. Techniques like LoRA train only a small set of extra weights, so a 7-billion-parameter model can be adapted on a single GPU. For most business tasks, though, good prompting or retrieval is cheaper and should come first.

People compare pairs of answers and pick the better one. In RLHF, reinforcement learning from human feedback, a reward model learns those preferences and the LLM is trained to score well on it. Newer methods such as DPO learn from the preference pairs directly. Some labs also use AI feedback guided by written principles.

Recent reasoning models add reinforcement learning on tasks with checkable answers, like maths and code, so the model learns to work through problems step by step before answering.

Because the core skill is prediction, a model can produce fluent text that is false. Its knowledge stops at a training cutoff. And its tone and refusals come mostly from fine-tuning and preference tuning, not from pretraining.