Lesson 183 · 3 min
Measuring whether an AI product works
An AI feature works when it helps people finish their real task, so measure outcomes like time saved and problems solved, beyond how impressive the answers sound.
It is easy to measure the wrong thing. "Users sent 10,000 messages" might mean the bot is popular, or that it took ten messages to get one useful answer.
Start from the job. For a help bot: what share of questions were solved without a human, and were customers happy afterwards? For a writing helper: how much of the draft did people keep, and how long did the task take?
Combine numbers with quality checks. Your eval set tells you whether answers are correct. User signals such as thumbs up, edits and repeat visits tell you whether people find them useful. Reading a sample of real conversations shows the reasons behind both.
Compare against what came before. If you can, run an A/B test where some people get the AI feature and others do not, and compare the results. And watch for side effects, like faster replies that bring more complaints.
Do it now
Pick one outcome metric
- 1. Choose an AI tool you use at work or at home.
- 2. Write the real task it helps with in one sentence.
- 3. Decide one number that would show it is helping, and one that would warn you it is not.
I use an AI tool to [TASK]. Suggest one main success metric based on the outcome, two supporting signals, and one warning sign that the tool is causing new problems. Explain how a small team could track each.
Quick check
Keep going
Next →The cost of each AI questionWords from this lesson
- Eval
- A test you design to check whether an AI system does your specific job well, using examples with known good answers.
- A/B test
- Trying two versions of something with different groups of people to see which one works better.
- Benchmark
- A standard set of test questions used to score and compare AI models. High scores do not guarantee good results on your task.