Lesson 171 · 3 min
Testing an AI feature before launch
Before real users touch an AI feature, run it on a set of realistic test questions with known good answers, and score it the same way every time.
Normal software gives the same output for the same input, so a few tests go a long way. AI does not. The same question can get a different answer tomorrow, and a small change to the prompt can fix one case and break three others.
So teams build an eval set: fifty to a few hundred real-looking inputs with notes on what a good answer must include or avoid. Include easy cases, tricky ones, rude ones, questions in Vietnamese and English, and a few attempts to break the rules.
Run the whole set after every change and score the results. Some checks are automatic, like "did it include the opening hours". Others need a person, or a second model that grades against a written rubric, spot-checked by a person.
Then try to break it on purpose, which is called red teaming. Ask friends or colleagues to make it misbehave. Launch to a small group first, watch real conversations, and add every new failure to the eval set.
Do it now
Write ten test questions
- 1. Imagine an AI feature you would like, such as a bot that answers questions about your class or shop.
- 2. Write ten questions: five normal, three tricky, two that try to break the rules.
- 3. For each, write one line on what a good answer must include. You now have a tiny eval set.
I am testing an AI help bot for [PURPOSE]. Here are my test questions: [PASTE] Suggest five more that cover cases I missed, especially tricky or unfair ones, and for each say what a good answer must do.
Quick check
Words from this lesson
- Eval
- A test you design to check whether an AI system does your specific job well, using examples with known good answers.
- Red teaming
- Deliberately attacking your own AI system, the way a bad actor would, to find weaknesses before release.
- Benchmark
- A standard set of test questions used to score and compare AI models. High scores do not guarantee good results on your task.
- A/B test
- Trying two versions of something with different groups of people to see which one works better.