L066 · Under the hood · 2 min
How AI is tested, and what tests miss
Benchmarks are standard tests that let people compare AI models, but a high score does not guarantee the model is good at your job.
01 · The idea
A benchmark is a fixed set of questions with known answers: maths problems, coding tasks, exam questions, reading tests. Run a model on it and you get a score you can compare with other models.
Scores help, and they have limits. When models get close to full marks, a test stops telling them apart, so new, harder tests keep appearing. Test questions can also leak into training data, and then the model may simply remember the answers.
Most benchmarks also measure narrow skills. Writing a clear letter to an upset customer in Vietnamese, or following your company's style, rarely appears in any public test.
The best test is your own. Pick five real tasks from your work, run them on two models, and judge the results side by side. Builders call this an eval.
02 · Think of it like this
It is like choosing a driver by their written test score. Useful, but you would still want to sit in the car while they drive through your own streets.
03 · Do it now
04 · Quick check
Words in this lesson
See also
L171Under the hood · 3 min
Testing an AI feature before launch
Before real users touch an AI feature, run it on a set of realistic test questions with known good answers, and score it the same way every time.
L183Under the hood · 3 min
Measuring whether an AI product works
An AI feature works when it helps people finish their real task, so measure outcomes like time saved and problems solved, beyond how impressive the answers sound.
L159AI at work · 3 min
How to compare AI assistants fairly
To know which assistant suits you, give each the same real tasks, the same way, and judge the answers against a simple scorecard.
L166Under the hood · 2 min
When AI teams fail
Teams of agents fail in very human ways: vague briefs, lost messages, duplicated work and nobody checking the final result.