Lesson 172 · 2 min
The control room: watching an AI app after launch
Once an AI feature is live, the team needs a control room that shows answers, costs, speed and complaints, so problems are found before they spread.
Launch is when the real learning starts. Users ask things nobody tested, in ways nobody expected. Without monitoring, a team only learns about a bad answer when it appears on social media.
A good control room tracks a few things. How many questions come in, how long answers take, how much each one costs, how often guardrails fire, and how often users click thumbs down, ask for a human or give up.
It also keeps samples of real conversations, with personal data removed, so people can read what actually happened. Reading twenty chats a week often teaches more than any chart.
Set alerts for sudden changes, like costs doubling overnight or complaints spiking after an update. When the model provider releases a new version, watch closely, since behaviour can shift even when your own code did not change.
Do it now
Choose five dials
- 1. Imagine you run a help bot for a small business.
- 2. Pick the five numbers you would want on your control room screen.
- 3. For each, decide what change would make you take action, like "thumbs down above 10 percent".
I run an AI help bot for [BUSINESS]. Suggest a simple monitoring dashboard: the five most useful numbers, how to collect each, and the warning level for each. Keep it suitable for a team of two.
Quick check
Words from this lesson
- Eval
- A test you design to check whether an AI system does your specific job well, using examples with known good answers.
- Latency
- The delay between asking an AI something and getting the start of its reply.
- Guardrails
- Rules and filters around an AI system that block harmful, off-topic or unsafe requests and answers.
- Human in the loop
- Designing a process so a person reviews or approves the AI's work before it has real consequences.