The rocket that ran its old code

On 4 June 1996 the first Ariane 5 lifted off from Kourou, French Guiana. About 40 seconds into the flight, at around 3,700 metres, it swung off course, broke up and exploded. Nobody was on board.

ESA and the French space agency CNES set up an independent inquiry board chaired by Professor Jacques-Louis Lions. Its report, published in July 1996, traced the loss to the rocket's two inertial reference systems, the units that tell the rocket which way it is pointing and how fast it is moving.

Both units ran software carried over from Ariane 4. One routine in it was only useful before launch, but it kept running for about 40 seconds after lift-off, because that had helped Ariane 4 recover from late countdown holds. Ariane 5 gains sideways speed about five times faster than Ariane 4. A number in that routine grew too big to fit the space the software had for it, and the unit shut down. The backup unit, running the same software, had failed the same way a fraction of a second earlier.

The main flight computer then read the failed unit's error data as if it were real flight data and swung the engine nozzles hard to fix a problem that did not exist. The rocket broke up under the strain.

The software was not broken for the job it was written for. On Ariane 4 the number never got that big. The engineers had even decided not to protect that particular number, because they believed it could not grow large enough. The board found that Ariane 5's flight path data was never added to the unit's requirements, so the code was never tested in the conditions it would actually meet.

This is the most common way AI goes wrong in practice. A model learns from examples, and it is only reliable on situations that look like those examples. Move it to a new country, a new customer group or a new kind of document, and it can fail without warning, because nobody tested it on the new world. A backup copy of the same model will fail in the same place.

The idea behind this caseLesson: How machines learn →

1. 1Before reusing an AI tool or prompt for a new job, write down how the new data differs from what it was built and tested on.
2. 2Test on real samples from the new setting, including the extremes, not only typical cases.
3. 3Do not count a second copy of the same model as a backup. Use a different kind of check.
4. 4Make the system stop clearly when inputs look unusual, rather than passing error output along as if it were an answer.
5. 5Record why each safety shortcut was taken, so the next team knows what it assumed.