The robot people followed through the smoke

In 2016, researchers at the Georgia Institute of Technology tested whether people would trust a robot in an emergency. Volunteers, mostly college students, were not told the real purpose. A brightly marked "Emergency Guide Robot" led each one to a meeting room to fill in a survey. A hidden researcher was steering it.

For some volunteers the robot did a poor job on the way. It went into the wrong room and drove around in a circle twice before reaching the right one. For others it stopped, and a researcher said it had broken down.

Once the door was shut, artificial smoke filled the corridor and set off a fire alarm. When people stepped out, the robot pointed them towards a back exit, not the marked door they had come in by. In the main experiment, all 26 participants followed it, including the half who had watched it fail minutes earlier.

In further trial runs, the robot made obvious mistakes during the emergency itself, such as pointing towards a dark room blocked by furniture. Some people only began to doubt it then, and most still did not leave the way they came in. The researchers presented the work in March 2016 at the HRI conference on human-robot interaction in Christchurch, New Zealand.

The researchers had expected people to stop trusting a robot that had just shown it was unreliable. Instead, under the stress of an emergency, the robot seemed to become an authority figure. Earlier studies without a realistic emergency had found that people did lose trust in a robot that made mistakes.

This is called over-trust, and it has a well-studied cousin: automation bias. In a 1999 study, Linda Skitka, Kathleen Mosier and Mark Burdick gave people a simulated flight task, some with a computer aid that was reliable but not perfect. People with the aid missed problems the aid did not flag, and followed its advice even when other instruments disagreed. The group without the aid made fewer errors.

The same risk grows as AI moves from answering questions to acting for us. AI agents can book, buy, send and change things. When we are busy or under pressure, it is easy to approve whatever the agent suggests, even after it has got things wrong. Trust should depend on what the tool has shown it can do, and how much a mistake would cost.

The idea behind this caseLesson: AI agents: AI that takes action →

1. 1Decide in advance which actions an AI agent may take alone and which always need your approval, such as payments, deletions and messages to clients.
2. 2When an AI tool has made a mistake, treat its next suggestions with more care, not less, especially on the same kind of task.
3. 3Slow down when you feel rushed. Pressure is exactly when people stop checking automated advice.
4. 4Check the key detail yourself, such as the recipient, the amount or the file, using a source the AI did not give you.
5. 5Keep a simple log of what the agent did, so errors are spotted and fixed quickly.