Lesson 66 · 2 min
How AI is tested, and what tests miss
Benchmarks are standard tests that let people compare AI models, but a high score does not guarantee the model is good at your job.
A benchmark is a fixed set of questions with known answers: maths problems, coding tasks, exam questions, reading tests. Run a model on it and you get a score you can compare with other models.
Scores help, and they have limits. When models get close to full marks, a test stops telling them apart, so new, harder tests keep appearing. Test questions can also leak into training data, and then the model may simply remember the answers.
Most benchmarks also measure narrow skills. Writing a clear letter to an upset customer in Vietnamese, or following your company's style, rarely appears in any public test.
The best test is your own. Pick five real tasks from your work, run them on two models, and judge the results side by side. Builders call this an eval.
Do it now
Run a mini eval
- 1. Write down three tasks you really do, such as a reply email, a summary and a short translation.
- 2. Give the exact same prompt to two different chatbots.
- 3. Score each answer from 1 to 5 for accuracy and usefulness, without looking at which tool wrote it.
Quick check
Keep going
Next →Small models and large models