The automatic debt letters nobody checked

From 2015, the Australian government used an automated system, later nicknamed Robodebt, to find welfare overpayments. It took yearly income data from the tax office and spread it evenly across each two-week period, then compared it with what people had reported.

If the numbers did not match, the system could raise a debt, and people had to prove they did not owe it. Many had worked irregular hours, so the average was wrong for them. Nearly half a million people received debt notices.

In late 2019, the Federal Court found the scheme unlawful. The government withdrew about A$1.76 billion in debts and settled a class action in 2021. In July 2023, a Royal Commission called the scheme "crude and cruel" and found it caused severe harm, including links to suicides.

Robodebt was not modern AI. It was a simple automated rule. But it shows the same danger as AI used at scale: a calculation that looks precise is trusted, and human checks are removed.

Averaging assumes people earn the same amount every fortnight. Real lives are uneven. When an automated system's basic assumption is wrong, it makes the same mistake for everyone, very fast, and the burden falls on the people least able to fight back.

The idea behind this caseLesson: Review AI like a manager →

1. 1Check that any automated decision follows the law before it goes live, not after complaints.
2. 2Keep a trained person reviewing decisions that cost people money or rights.
3. 3Never make people prove a machine wrong; the organisation should show the evidence.
4. 4Test the system on real, messy cases, like people with irregular jobs.
5. 5Track complaints and errors, and stop the system quickly when warning signs appear.