Lesson 159 · 3 min
How to compare AI assistants fairly
To know which assistant suits you, give each the same real tasks, the same way, and judge the answers against a simple scorecard.
People often pick an assistant from one impressive screenshot or one bad answer. That is like choosing a restaurant from one dish on one night. Results vary from day to day and from task to task.
A fair test starts with your own work. Choose five tasks you really do, such as a customer reply, a summary of a long PDF, a Vietnamese translation, a spreadsheet question and a planning task. Write each prompt once and paste the exact same text into every assistant.
Before reading the answers, decide what good looks like: correct facts, follows instructions, right tone, saves you time. Score each answer from 1 to 3 on each point. If you can, hide which tool wrote which answer while you score.
Run each prompt twice, because answers change. Note the things scores miss too: privacy settings, price, language support and whether it works on your phone. The best assistant is the one that does your tasks well, whatever the leaderboards say.
Do it now
Run a five-task taste test
- 1. Write five prompts from your real work and save them in a note.
- 2. Paste each into two or three assistants and copy the answers into a document without names.
- 3. Score them on accuracy, instructions followed and usefulness, then reveal the names.
Quick check
Keep going
Next →What tools and plugins let AI doWords from this lesson
- Benchmark
- A standard set of test questions used to score and compare AI models. High scores do not guarantee good results on your task.
- Eval
- A test you design to check whether an AI system does your specific job well, using examples with known good answers.
- Bias
- When an AI treats some people or ideas unfairly because of skewed patterns in the data it learned from.