The risk score that wrecked families

For years, the Dutch tax authority used a risk-scoring system to flag childcare benefit claims as possibly fraudulent. According to Amnesty International, not having Dutch nationality raised a person's risk score.

Tens of thousands of parents were wrongly accused of fraud. Many had to repay large sums, which led to debt, job losses and broken families. The tax authority admitted that at least 11,000 people were singled out for extra checks because of their ethnic origin or dual nationality.

A parliamentary report in December 2020 called it an unprecedented injustice. On 15 January 2021, Prime Minister Mark Rutte's government resigned over the scandal. Amnesty's October 2021 report, Xenophobic Machines, examined the algorithm's role.

A risk score is only as fair as the signals put into it. Using nationality as a signal built discrimination into the design. Amnesty also reported that the system learned and adapted over time, which repeated the same bias.

Officials checking flagged cases were not told why the system had given someone a high score. When a machine's reasoning is hidden, people tend to trust it. Nobody could easily spot or challenge the error.

The idea behind this caseLesson: Review AI like a manager →

1. 1Never use protected traits like nationality or ethnicity as risk signals, and check for stand-ins like postcode or name.
2. 2Make every automated flag explainable: the reviewer must see why a case was flagged.
3. 3Treat a flag as a reason to look closer, not as proof of guilt.
4. 4Give people a clear, fast way to see and challenge the decision about them.
5. 5Have an independent team audit outcomes by group, regularly and in public.