The marshmallow that predicted less than people thought

From the late 1960s, the psychologist Walter Mischel and his colleagues ran studies at a nursery school on Stanford University's campus. A child aged about four or five sat at a table with a treat, such as a marshmallow or a pretzel. The researcher explained that they could eat it now, or wait until the researcher came back and get two. Then the researcher left the room for up to about 15 to 20 minutes.

The early papers, including a well-known one in 1972, were about how children waited. Those who distracted themselves, sang, covered their eyes or thought about the treat as a picture waited longer than those who stared at it.

The fame came later. Follow-up surveys of some of the same children, published in 1988 and 1990, found that those who had waited longer tended to have higher SAT scores and better ratings from parents as teenagers. The test became a popular symbol of willpower and a supposed predictor of success.

In 2018, Tyler Watts, Greg Duncan and Haonan Quan repeated the follow-up idea with roughly 900 children from a larger and more varied group of families. The link between waiting and later school results was about half as strong as in the original reports, and it shrank much further once they accounted for family background and early thinking skills.

The first lesson is about who was in the sample. The original children came from families connected to a wealthy university, and the follow-ups reached only some of them. A pattern found in a narrow group can look much stronger than it is.

The second is about hidden causes. A 2013 study by Celeste Kidd and colleagues found that children waited far longer when the adult had kept an earlier promise than when the adult had broken one. For a child who has learned that promised treats often vanish, eating now is a sensible choice. Waiting partly measured trust and home life, which also shape later results.

Machine learning has the same trap. A model trained on one group finds patterns that may not hold elsewhere, and a feature that seems to predict an outcome may only stand in for something else, like income or postcode.

The idea behind this caseLesson: How machines learn →

1. 1When a single test is said to predict a big outcome, ask who was studied and how many.
2. 2Look for a hidden third factor, such as money, age or location, that could drive both things.
3. 3Prefer findings repeated in larger and more varied groups before you act on them.
4. 4When an AI tool scores people, ask what data it learned from and whether that data looks like the people you serve.