What attention actually does inside an LLM

Every large language model in production today does exactly one thing: given a sequence of tokens, it produces a probability distribution over which token comes next. Everything else — reasoning, code, translation, refusal — is behaviour that fell out of doing that one thing well enough, at enough scale, on enough text.

It is worth sitting with how strange that is. There is no plan for the sentence. The model does not know what its own third word will be when it emits the first.

Tokens are not words

Text is first split into tokens — usually sub-word fragments. Common words are single tokens; rarer ones are assembled from pieces. This is a compression decision, and it leaks: it is the reason models are historically poor at counting letters in a word or doing arithmetic on long numbers. They never saw the letters. They saw a token.

The problem attention solves

Consider: "The city fixed the pothole it reported." What does "it" refer to? A human resolves this instantly. A model processing tokens in isolation cannot — the token for "it" carries no information about which earlier noun it stands for.

Attention is the mechanism that lets every token look at every other token and decide which ones matter for interpreting itself.

Queries, keys and values

Each token produces three vectors. A query — what am I looking for. A key — what I can offer. A value — what I actually contribute if selected.

Every query is compared against every key by dot product, producing a score for each pair. The scores are scaled and passed through a softmax, which turns them into weights that sum to one. Each token's new representation is then the weighted sum of all the values.

That is the entire operation. Its power comes from being learned: nothing tells the model that pronouns should attend to nouns. The weights that produce the query, key and value vectors are trained by gradient descent like any other weights, and this behaviour is what minimised loss.

• Multi-head attention runs several of these in parallel, each free to specialise — one head tracking syntax, another long-range references.
• Self-attention is quadratic in sequence length: doubling the context quadruples the comparisons. This is the real constraint behind context-window limits.
• Causal masking stops a token attending to tokens that come after it, which is what makes the model a predictor rather than a text autocompleter with hindsight.

Generation, one token at a time

To produce text, the model runs the whole sequence forward, gets a distribution over the next token, samples one, appends it to the input, and runs again. Temperature controls how sharply it favours the most likely token — low is repetitive, high is erratic.

Why they make things up

Hallucination is not a bug that will be patched. The training objective rewards producing plausible continuations, and a fluent wrong answer scores better against that objective than an admission of ignorance — because text confidently stating things is what the training data overwhelmingly contains.

Fine-tuning and reinforcement from human feedback reduce this considerably. They do not remove it, because they do not change what the underlying model is optimising for. Retrieval — putting the actual source documents into the context — helps far more, since it converts a recall problem into a reading problem.

Where this leaves you

A language model is an extraordinary tool for anything where the answer can be checked, and a liability anywhere it cannot. That distinction is the whole of practical LLM engineering.

Ask it to draft, summarise, translate or restructure — work you can verify at a glance. Do not ask it to be the system of record.

For civic infrastructure this is decisive. A model that classifies a photograph and routes it, where the classification is checkable and a human sees the outcome, is a sound use. A model that answers a resident's question about their legal rights, unverified, is not.