ManuAIVI

L004 · How AI thinks · 1 min · Interactive

Tokens: how AI reads

AI does not read words; it reads tokens, small chunks of text that are often parts of words.

01 · The idea

Before a model can read your message, the text is cut into tokens. A common English word like "the" is one token. A long or rare word, like "unbelievable", may be split into two or three.

Languages are not treated equally. The same sentence in Vietnamese, Hindi or Amharic often uses more tokens than in English, because the model saw less of those languages in training.

Tokens matter for three reasons. Paid AI services charge per token. Every model has a limit on how many tokens it can handle at once. And odd token splits explain strange mistakes, like miscounting the letters in a word.

Try it below: type a sentence and watch it split into tokens. Try your own language too.

02 · Think of it like this

Think of Lego. The model does not see a finished house; it sees the bricks. Common words are one big brick, rare words are built from several small ones.

03 · Try the toy

Try it

See text the way an AI does

Type anything, or tap a sample. Each coloured chip is one token, the unit an AI reads and is billed by.

Characters
86
Tokens
22
≈ Words
13
The quick brown fox jumps over the lazy dog. Tokenization is surprisingly unintuitive!

That's about 1.7 tokens per word. Plain English usually lands near 1.3; accents, other alphabets, emoji and code cost more, so the same message can fill an AI's memory (and your bill) faster.

Approximate: real tokenizers differ from model to model.

04 · Do it now

05 · Quick check

Why might a chatbot struggle to count the letters in "strawberry"?

Words in this lesson

See also

Play it

Lab · 10 min · runs in browser

AI Brain Lab

Watch a real tokenizer cut up your text, build a tiny next-word machine, and spin a temperature wheel.

L004Interactive