The dealer bot that "sold" a car for $1

In December 2023, a California car dealer, Chevrolet of Watsonville, had a chatbot on its website, supplied by a company called Fullpath and powered by ChatGPT. A visitor noticed and asked it to write computer code. It did.

Others joined in. One visitor gave the bot new instructions, including ending its replies with "and that's a legally binding offer". He then asked for a 2024 Chevy Tahoe for $1, and the bot agreed.

The screenshots went viral. The offer was not legally binding. Fullpath shut down the bot on that dealer's site and said the bot had resisted most attempts and revealed no confidential data.

A chatbot follows instructions written in plain language. The company's instructions and the visitor's messages arrive as the same kind of text. A clever visitor can write instructions that override the company's. This is called prompt injection.

The bot was a general-purpose model with few limits on what it could talk about or promise. Nothing stopped it from making statements about prices that only a human salesperson should make.

The idea behind this caseLesson: AI agents: AI that takes action →

1. 1Limit a customer-facing bot to a narrow job, like opening hours and booking appointments.
2. 2Block the bot from making offers, prices or promises; send those questions to a person.
3. 3Before launch, have someone try to trick the bot on purpose, and fix what they find.
4. 4Show a clear notice that the bot cannot make binding offers.
5. 5Watch the logs after launch and be ready to switch the bot off quickly.