Transformer

Text is split into tokens and each token becomes a vector. The model then runs those vectors through repeated layers of attention, where every token computes how relevant every other token is to it, and produces a new representation weighted by that relevance. Stack enough of those layers and the representation of a word ends up carrying the context it appeared in.

The word "bank" starts as one vector regardless of meaning. After a few attention layers in "river bank", it has absorbed information from "river" and no longer resembles the "bank" in "bank account". That is the whole idea, repeated at scale.

What it replaced processed text strictly in order, one word after another, which made training slow and made long-range connections fade. Attention looks at everything simultaneously, so training parallelises across thousands of chips and a word can connect to something a thousand words earlier just as easily as to its neighbour.

That parallelism is the reason models got large enough to be useful. The architecture did not just work better — it made spending enormous amounts of compute worthwhile, which is what produced the capabilities people now take for granted.

That attention is free. Comparing every token to every other token means cost grows with the square of the input length: doubling your context roughly quadruples the attention work. This is why long-context requests get expensive and slow rather than just slightly slower, and why filling a context window because you can is a bad habit.

The second is thinking the architecture explains the behaviour. Transformers are the shape of the thing, but capability comes from scale, data quality and the training that follows. Two models with identical architecture can be worlds apart.