Sequences, attention and transformers

Words come in order, and meaning depends on that order. Models first split text into tokens, small pieces of words, and turn each token into a vector of numbers called an embedding.

Older models, recurrent neural networks (RNNs) and LSTMs, read tokens one at a time and carried a running memory forward. They worked, but they were slow to train because each step waited for the previous one, and they forgot details from far back in long texts.

Attention lets every token look at every other token at once and decide how much each one matters to it. In "The trophy did not fit in the suitcase because it was too big", attention helps "it" connect strongly to "trophy".

Mechanically, each token makes a query, a key and a value. Matching queries against keys gives scores. The scores become weights, and the token's new meaning is a weighted mix of the values. Several attention heads run side by side, each free to track a different kind of relation.

The transformer, introduced in the 2017 paper "Attention Is All You Need", stacks blocks of attention and small feed-forward layers. Because it has no step-by-step loop, the whole sequence can be processed in parallel on GPUs. That is what made training on huge amounts of text practical.

Attention has no built-in sense of order, so position information is added to the embeddings. And comparing every token with every other means cost grows with the square of the length, which is one reason context windows have limits.

Decoder-only transformers that predict the next token power chatbots like ChatGPT, Claude and Gemini. Encoder models like BERT are used for search and classification. The same idea, applied to image patches or audio, drives vision and speech models too.