The chatbot that learned abuse in a day

On 23 March 2016, Microsoft launched Tay, a chatbot on Twitter aimed at 18 to 24 year olds. Microsoft said the more people chatted with Tay, the smarter it would get.

Within hours, groups of users began feeding it offensive messages. Tay started posting racist, sexist and inflammatory tweets, often by repeating what users had told it to say.

Within its first day, Microsoft took Tay offline and began removing the worst tweets. On 25 March, it apologised, saying a coordinated attack by some users had exploited a weakness it had not foreseen.

Tay learned from the people talking to it. That sounds friendly, but it means anyone can become a teacher, including people who want it to fail. Feeding a system bad examples on purpose is a known attack.

Microsoft said it had planned filters and tests, but missed this specific attack. A public system that keeps learning from strangers needs strong limits on what it can learn and repeat.

The idea behind this caseLesson: How machines learn →

1. 1Do not let a public AI system learn directly from live users without review.
2. 2Block "repeat after me" style features and filter what the system can say in public.
3. 3Have a team try to abuse the system before launch, thinking like trolls.
4. 4Launch to a small group first and watch closely.
5. 5Keep a fast off-switch and a plan for cleaning up.