The horse that seemed to do sums

In the early 1900s, a retired mathematics teacher in Berlin, Wilhelm von Osten, showed off his horse, Hans. Asked to add numbers, count the people in the room or name a playing card, Hans tapped out the answer with his right front hoof. Crowds and newspapers were amazed.

In September 1904 a panel of experts, led by the psychologist Carl Stumpf, watched Hans and found no sign of deliberate trickery. Stumpf's colleague Oskar Pfungst then ran controlled tests. The key one was simple: sometimes the person asking knew the answer, and sometimes nobody in the room did.

The results were stark. When the questioner knew the answer, Hans was right almost every time. When nobody knew, he was nearly always wrong. With large blinkers on, Hans got 89% right in tests where he could still see the questioner, and only 6% when he could not.

Pfungst found the cue. As Hans started tapping, the questioner leaned forward slightly. When the right number was reached, the person relaxed or gave a tiny upward jerk of the head, and Hans stopped. Nobody was cheating. Even von Osten did not know he was doing it. Pfungst published his findings in 1907.

Hans was not stupid. He had learned a real skill: reading people. But it was not the skill everyone thought he had. Whenever the easy signal was removed, the apparent ability vanished.

AI models do the same thing, and researchers now call it the Clever Hans effect. A model learns whatever pattern best predicts the right answer in its training data, even if that pattern is a shortcut. In a 2019 study, Sebastian Lapuschkin and colleagues found an image classifier that recognised horses partly by a copyright tag that appeared on many horse photos in its dataset. When they pasted that tag onto a picture of a car, the model's answer switched to horse.

A high score on a test does not tell you how the model got there. If the test data shares the same shortcut as the training data, the shortcut looks like intelligence until the model meets the real world.

The idea behind this caseLesson: How machines learn →

1. 1Test an AI tool on fresh cases that look different from the examples it was built or tuned on, not only on similar ones.
2. 2Run a blind check: give it questions where you know the answer but nothing on the surface gives that answer away.
3. 3Ask what else differs between the groups you want it to tell apart, such as lighting, file names, dates or who took the photo, and remove those clues.
4. 4Where the tool can show what it looked at, check that it focused on the thing that matters, not on a corner or a watermark.
5. 5Be suspicious of results that look too good, and check them again before you rely on them.