What the model actually does

A language model does exactly one thing: given a sequence of tokens, it produces a probability for every token that could come next. That single sentence, taken seriously, explains most of what you will meet in this pathway — the abilities and the failure modes alike. This stage unpacks it far enough that nothing downstream is mysterious.

Tokens, embeddings, attention

Tokens, not words: text is chopped into common fragments (roughly ¾ of an English word each, worse for code and non-English text), which is why models are billed and limited in units that do not map cleanly onto words — and why odd arithmetic and spelling failures often trace to tokenisation rather than reasoning. Each token becomes a vector — Foundations’ maths, employed — and attention is the operation that lets every position ask, in parallel, “which earlier tokens matter for predicting what comes after me?” Mechanically it is learned projections (queries, keys, values), a similarity score, a softmax, a weighted sum — matrix multiplications all the way down. Stack a few dozen layers of attention and feed-forward blocks, and you have the transformer. That its cost grows with context length is not an implementation detail; it is why the rest of this pathway is largely about spending context well.

Where the time and money go

Generation runs in two phases with different economics, and knowing them is how you reason about latency bills:

Prefill processes the whole prompt in one parallel pass and produces the KV cache — the attention state for everything read so far. It sets time-to-first-token, and it grows with prompt length: a 100k-token context is paid for here, in latency, money and GPU memory. Decode then produces one token per step, each step consulting the cache — which is why output streams, why long answers cost linearly, and why input tokens are priced differently from output tokens. The practical reflexes follow directly: stream the decode (users read t1 while t9 is being made), keep prompts as short as correctness allows, and put the stable parts of your prompt first — providers cache identical prefixes, so a fixed system prompt followed by variable content is dramatically cheaper than the reverse.

Sampling, and the model’s two boundaries

The raw output is a distribution; sampling turns it into text. Temperature rescales scores before softmax — low sharpens toward the favourite (use ~0 for extraction and structured output), high flattens toward variety (drafting, brainstorming). Top-p truncates the long tail of unlikely tokens before sampling. This is also the honest explanation of “the model said something different this time”: you drew twice from a distribution.

Two boundaries govern everything. The weights are frozen: what the model “knows” was fixed at training (the knowledge cutoff), and changing weights — fine-tuning — is a project, not a request. The context window is its entire working memory: it knows nothing about your request except the tokens you assembled. Between frozen weights and assembled context, all of your day-to-day leverage is the context — which is why the standard escalation ladder is prompting → retrieval → fine-tuning, in that order, each step only when the previous one measurably fails. Hallucination stops being mysterious from this vantage: the machine’s only move is “emit a plausible continuation”, and when truth is weakly represented, plausible and true part ways — fluently. Not lying; doing the one thing it does, on the wrong material.