Lesson 31 · 2 min
Attention, in plain words
Attention is how a model decides which earlier words matter most for the word it is writing now.
When a chatbot reads your message, it breaks it into tokens. To understand any one token, it looks back at all the others and asks: which of these help me make sense of this one?
Take the sentence "The cat sat on the mat because it was warm." To work out what "it" means, the model gives more weight to "mat" and "warm" and less to "the" or "on". That weighting is called attention.
Modern language models are built on a design called the transformer, and attention is its core trick. The model runs many attention steps in layers, each one picking up different links: grammar, meaning, who did what to whom.
This is why chatbots can follow long, messy messages. It also explains a weakness: attention gets spread thin over very long text, so small details can get less focus.
Do it now
Watch attention at work
- 1. Write a sentence where a word like "it" or "they" could mean two things.
- 2. Ask a chatbot what the word refers to, using the prompt below.
- 3. Change one word in the sentence and ask again. See if its answer follows the change.
In this sentence, what does "it" refer to, and which other words told you? Sentence: "The trophy did not fit in the suitcase because it was too big."
Quick check
Keep going
Next →Why AI forgets in long chatsWords from this lesson
- Attention
- The part of a transformer that lets each word weigh how much every other word in the text matters to its meaning.
- Transformer
- The neural network design behind modern chatbots, good at tracking how every word in a passage relates to the others.
- Token
- A chunk of text, often part of a word, that AI reads and writes in. Limits and prices are counted in tokens.