The machine that said the patient got almost nothing

The Therac-25 was a radiation therapy machine made by Atomic Energy of Canada Limited (AECL). Earlier models had physical safety locks that stopped the beam if something was set up wrong. The Therac-25 relied much more on its software to do that job.

Between June 1985 and January 1987, the machine gave six known massive overdoses at clinics in the United States and Canada. When doctors and physicists raised the alarm, AECL said the machine could not overdose a patient and pointed to other causes.

In March 1986, at a cancer centre in Tyler, Texas, an operator typed the wrong treatment mode, quickly corrected it, and started the treatment. The screen showed "Malfunction 54" and a dose of 6 units out of the 202 prescribed. She pressed the key to proceed. The patient had in fact received a huge overdose. He died five months later. A second Tyler patient was hit in the same way in April and died in May. A patient in Yakima, Washington, overdosed in January 1987 because of a different software bug, died that April from complications.

The Tyler hospital's physicist, Fritz Hager, worked with the operator until they could make the error happen on purpose: it only appeared when the operator edited the settings fast. In May 1986 the US Food and Drug Administration declared the machine defective and made AECL produce a corrective plan, which ran to more than 20 hardware and software changes.

The fast-editing bug was a timing problem. If the operator changed the mode while the machine was still setting itself up, the software did not notice the change and used the wrong settings. The old physical locks would have stopped the beam regardless of what the software thought. On the Therac-25 there was nothing outside the software to catch its mistake.

The people around it had also learned to trust it. Error messages were cryptic and frequent, so pressing "proceed" became routine. The maker's safety analysis had looked at hardware failures and treated software errors as almost impossible.

Today's AI agents raise the same question. When software can act directly, sending money, changing records, emailing customers, the safety limit must not live only inside the same system that can be wrong. And an error message that people learn to click past is no longer a safety feature.

The idea behind this caseLesson: AI agents: AI that takes action →

1. 1Put hard limits outside the AI: spending caps, approval steps and permissions it cannot change by itself.
2. 2Treat any unexplained error from an AI tool that acts on real things as a reason to stop, not to retry.
3. 3Give every alert a plain-language meaning and a written next step, so nobody learns to click past it.
4. 4When a user reports harm, assume it may be real and investigate it, even if the vendor says it cannot happen.
5. 5Keep a log of what the AI did, so you can reconstruct a bad outcome step by step.