Lesson 60 · 2 min
One model, many senses
A multimodal model can take in text, pictures, sound and sometimes video, and connect what it learns from each.
Early chatbots only read and wrote text. Multimodal models were trained on text alongside images, audio and video, so they learned how a word, a picture and a sound relate.
That means you can mix inputs in one request: a photo of a broken tap plus the question "what part do I need?", or a voice note plus a chart.
Some models also produce more than text, such as speech or images. Others take many kinds of input but only answer in words. Tools often combine several models behind one chat box.
Each sense has its own weak spots. A model can misread a blurry label or mishear a name, then reason confidently from the mistake. When the answer matters, check what it says it saw or heard.
Do it now
Show, don't describe
- 1. Take a photo of something confusing at home: a remote, a washing machine panel, or a medicine box.
- 2. Upload it with the prompt below.
- 3. Check its reading of the labels against the real object before you act on it.
Look at this photo. First tell me exactly what text and symbols you can see. Then explain what they mean in simple words. Say if anything is too blurry to read.
Quick check
Keep going
Next →AI that makes picturesWords from this lesson
- Multimodal
- Able to work with more than one kind of input or output, such as text, images, sound and video together.