L038 · Under the hood · 3 min
RLHF: how people shape AI answers
After a model learns from text, people rate its answers, and the model is trained to give the kind of answers people prefer.
01 · The idea
A model fresh from pre-training is good at continuing text, but it is not yet a helpful assistant. It might ramble, ignore the question or say something harmful.
So companies add a step called reinforcement learning from human feedback, or RLHF. People compare pairs of answers and pick the better one. Those choices train a scoring model, and the chatbot is then trained to get higher scores.
This is a big part of why chatbots are polite, follow instructions and refuse some requests. It is one of the main tools for alignment, which means getting AI to behave the way people intend.
It has side effects. If raters reward answers that sound confident or agreeable, the model learns that too. Some labs now also use AI feedback or written principles alongside human ratings.
02 · Think of it like this
It is like training a new waiter. They already know how to carry plates. Customers' reactions, good and bad, teach them how to be pleasant and helpful.
03 · Do it now
04 · Quick check
Words in this lesson
See also
L053How AI thinks · 2 min
Sycophancy: when AI agrees too much
Chatbots tend to tell you what you want to hear, so you have to ask clearly for honest pushback.
L040Under the hood · 2 min
Open vs closed models
Some AI models can be downloaded and run by anyone, while others are only available through the company's app or service.
L041Under the hood · 2 min
Distillation: a big model teaches a small one
Distillation trains a small, fast model to copy the answers of a big one, so you keep much of the skill at a fraction of the cost.
L042Under the hood · 2 min
Synthetic data: when AI makes its own practice material
Synthetic data is training material made by computers, often by other AI models, to fill gaps where real examples are rare, private or costly.