04
Lesson 4 · 1 min
Tokens: how AI reads
AI does not read words; it reads tokens, small chunks of text that are often parts of words.
Before a model can read your message, the text is cut into tokens. A common English word like "the" is one token. A long or rare word, like "unbelievable", may be split into two or three.
Languages are not treated equally. The same sentence in Vietnamese, Hindi or Amharic often uses more tokens than in English, because the model saw less of those languages in training.
Tokens matter for three reasons. Paid AI services charge per token. Every model has a limit on how many tokens it can handle at once. And odd token splits explain strange mistakes, like miscounting the letters in a word.
Try it below: type a sentence and watch it split into tokens. Try your own language too.
See text the way an AI does
Type anything, or tap a sample. Each coloured chip is one token, the unit an AI reads and is billed by.
- Characters
- 86
- Tokens
- 22
- ≈ Words
- 13
That's about 1.7 tokens per word. Plain English usually lands near 1.3; accents, other alphabets, emoji and code cost more, so the same message can fill an AI's memory (and your bill) faster.
Approximate: real tokenizers differ from model to model.
Do it now
Count the cost of a language
- 1.Type one sentence in English into the tokenizer above.
- 2.Type the same sentence in another language you know.
- 3.Compare the token counts. The bigger number costs more and fills the model's memory faster.
Words from this lesson
- Token
- A chunk of text, often part of a word, that AI reads and writes in. Limits and prices are counted in tokens.
- LLM
- An AI trained on huge amounts of text to predict the next word, which lets it write, summarise, translate and chat.
- Context window
- The maximum amount of text an AI can look at in one go. Anything beyond it is forgotten for that conversation.