The hiring tool that learned to prefer men

From 2014, a team at Amazon built software to score job applicants from one to five stars, so recruiters could find top candidates faster. The model learned from CVs sent to the company over the previous ten years.

By 2015, the team saw that it was not rating candidates for technical jobs in a gender-neutral way. According to Reuters, it penalised CVs that included the word "women's", as in "women's chess club".

Engineers edited the model to ignore those terms, but could not be sure it would not find other ways to discriminate. The project was dropped. Reuters reported the story in October 2018, adding that recruiters had looked at the tool's suggestions but never relied on them alone.

Machine learning finds patterns in past examples. Most of the CVs, and most of the people hired, in that tech-heavy history were men. So the model learned that signs of being a woman went with "not hired", and treated them as bad signs.

Removing one word does not remove the pattern. A model can pick up the same signal from other clues, like a club, a college or a choice of words. Bias in the past data becomes bias in the predictions.

The idea behind this caseLesson: How machines learn →

1. 1Before training or buying a model, ask what past data it learned from and who is missing from that data.
2. 2Test results across groups (for example by gender or age) before use, and keep testing after launch.
3. 3Use AI scores as one input for a human decision, never as an automatic filter.
4. 4Ask vendors for evidence of bias testing, in writing.
5. 5Give candidates a way to ask for a human review.