Lesson 65 · 2 min
The black box problem
Even the people who build large AI models cannot fully explain why a model gave a particular answer.
A neural network is billions of numbers adjusted during training. Nobody writes the rules by hand. The behaviour comes out of all those numbers working together.
So when a model gives an answer, there is no simple list of reasons to read. You can ask it to explain itself, but its explanation is just more generated text, and it may not match what really happened inside.
Researchers are working on this, in a field called interpretability. They have found ways to spot some ideas inside models and see how certain answers form. It is real progress, and still far from a full picture.
For now, this is why decisions about loans, jobs, health or the law should not rest on an AI answer alone. People need to be able to question a decision and get a reason a human can stand behind.
Do it now
Ask for a reason, then test it
- 1. Ask a chatbot to choose between two options, such as two phones or two holiday spots, and explain why.
- 2. Then swap the order you list them in and ask again in a new chat.
- 3. See whether the choice or the reasons change. That gap shows the limits of its self-explanation.
Quick check
Words from this lesson
- Explainability
- How well people can understand why an AI produced a particular answer or decision.
- Neural network
- Layers of simple maths units, loosely inspired by brain cells, whose connections are adjusted during training to find patterns.