Attention Is All You Need

The problem: reading one word at a time

Until 2017 the best models of language were recurrent: they read a sentence left to right, carrying a hidden state that summarised everything so far, one step per word. That has two costs. Training cannot be parallelised across the sentence, because step ten needs step nine, so the biggest models took weeks on the fastest hardware. And a word at the end of a long sentence has to reach a word at the start through every state in between; the signal fades, and the model forgets. An add-on called attention, introduced for translation in 2014, let the decoder peek back at the whole input, and it helped so much that the authors of this paper asked whether the recurrence was needed at all.

The idea: let every word look at every other word, in parallel

The Transformer has no recurrence and no convolution. Each layer is a self-attention step, in which every token builds a new representation of itself by looking at all the tokens in the sequence and taking a weighted mix of them, with weights it computes on the fly, followed by a small feed-forward network applied to each token separately. Because nothing in a layer depends on the token before it, the whole sentence is processed at once, and because every token can attend to every other directly, distance no longer matters. Position is reinjected by adding a fixed pattern to each token’s vector, so the model can still tell first from last.

How it works

From each token’s vector the model computes three smaller vectors by multiplying with learned matrices: a query, a key and a value. The score between two tokens is the dot product of one’s query with the other’s key, scaled down by the square root of the vector size so the softmax stays well-behaved. Softmax across a token’s scores gives its attention weights, and the token’s new vector is the weighted sum of the values. That whole computation is a couple of matrix multiplications, which is exactly what a GPU is for.

The rest of the architecture is support. Multi-head attention runs eight such lookups side by side so that different heads can learn different relations; residual connections and layer normalisation keep gradients healthy through many layers; the decoder masks future positions so it can only attend to what it has already generated. The paper’s base model trained in twelve hours on eight GPUs and beat the best translation systems of the day; the large model trained in three and a half days and set new records on both of its benchmarks.

What it changed

Within eighteen months the Transformer had replaced recurrent networks almost everywhere. BERT used the encoder for understanding; GPT used the decoder for generation; T5 used both. Then it left language: Vision Transformers cut images into patches and treated them as tokens, AlphaFold 2 used attention over amino acids, Whisper used it for speech. Every large language model since is a scaled-up decoder stack from this paper, and the title’s claim, provocative in 2017, is now a description of the industry.

Where you meet it today

• Every LLM, image generator and speech model you have used since 2020.
• The context window. Attention compares every token with every other, so its cost grows with the square of the length; that is why context length is a headline feature and an engineering problem.
• The KV cache. Keys and values of earlier tokens are stored during generation so they are not recomputed; serving cost and “context engineering” both come down to managing it.
• Attention maps as explanations, imperfect but useful, when someone asks what a model was looking at.