The test that asked a better question

In October 1950 the philosophy journal Mind published a paper by Alan Turing called "Computing Machinery and Intelligence". It opens: "I propose to consider the question, 'Can machines think?'" Turing then set that question aside, because words like "think" are too vague to settle by argument.

In its place he described what he called the imitation game. A judge sits in another room and exchanges typed messages with two hidden players, one a person and one a machine. The judge must work out which is which. If the machine is hard to tell apart from the person, Turing suggested, we should take that seriously.

Turing made a cautious prediction. In about fifty years, he thought, a computer could play well enough that an average judge would pick correctly no more than 70 percent of the time after five minutes of questions. He also went through a list of objections, such as the view of Ada Lovelace that a machine can only do what we tell it to, and answered each one.

Electronic computers were brand new. Turing was working with one of the first, at the University of Manchester. Nothing he could use came close to playing the game.

Turing's move was to judge a machine by what it does in a conversation, not by what is going on inside it. That idea became the most famous test in AI, now called the Turing test. It also left a gap that is still argued about: passing for human in a chat is not the same as understanding, knowing or caring.

For a chatbot user today this matters a lot. Modern chatbots are very good at the imitation game. They write fluent, friendly, human-sounding text. That fluency is exactly what Turing's test measures, and it tells you little about whether an answer is true. A chatbot can sound like a thoughtful person while it predicts likely words, one after another.

The idea behind this caseLesson: What a chatbot really does →

1. 1Judge a chatbot answer by checking it, not by how natural or confident it sounds.
2. 2Remember that "sounds human" and "is right" are two different tests.
3. 3When someone claims an AI "thinks" or "understands", ask what it actually did and how that was measured.
4. 4Use the chatbot's fluency for what it is good at, such as drafting and explaining, and keep your own judgement for decisions.