Serving and inference economics
Between the model and your users sits a serving stack with its own physics; its numbers set your product’s.
Between a trained model and a user sits a serving stack with its own physics, and its numbers — time-to-first-token, tokens per second, cost per million — set your product’s numbers. The transformer chapter introduced prefill and decode; this one treats serving as the systems problem it is, because whether you buy inference or host it, you are budgeting against these mechanics.
Two phases, two different machines
Prefill — reading the prompt — is one huge parallel pass: compute-bound, GPU flat out, and the source of time-to-first-token. Decode — producing tokens — is one small step per token that must re-read the entire KV cache each time: memory-bandwidth-bound, the GPU’s arithmetic mostly idle waiting on its own memory. Foundations’ ladder, at datacentre scale. This asymmetry explains the pricing you see everywhere (input tokens cheap, output tokens dear) and the two levers that matter most in practice: shorten what must be generated, and cache what must be read.
How serving stacks earn their keep
A naive server runs one request at a time and wastes the machine. Production stacks (vLLM and kin) fix utilisation with three moves. Continuous batching: decode steps from many requests share each GPU pass, and new requests join mid-batch rather than waiting for stragglers — throughput becomes a scheduling problem. Paged KV cache: cache memory is allocated in pages like an OS, not in worst-case contiguous slabs — concurrency stops being limited by fragmentation (yes, that is literally the virtual-memory chapter, replayed on a GPU). Prefix sharing: requests that begin with the same system prompt share its KV cache, so the stable prefix is computed once — the mechanism behind provider-side prompt caching, and the reason your prompts should keep their stable parts first.
Making the model cheaper
| Technique | What it does | What it costs |
|---|---|---|
| Quantisation | weights at 8-bit or 4-bit instead of 16 — less memory bandwidth, faster decode | small quality loss; measure on your evals, not the leaderboard |
| Distillation | a small model trained to imitate a large one on your task | training effort; narrower competence |
| Speculative decoding | a draft model proposes several tokens; the big model verifies in one pass | complexity; wins depend on acceptance rate |
| Routing | easy requests to a small model, hard ones to a large | a classifier to build and evals to justify it |
Buy or host — the honest spreadsheet
APIs win on zero ops, instant scale and frontier quality; hosting wins only when a column of the spreadsheet says so: sustained high utilisation (idle GPUs are the most expensive silence in computing), data that legally cannot leave, latency floors APIs cannot meet, or an open model your evals show is genuinely sufficient. Do the arithmetic in tokens per second per dollar at your utilisation — not at 100% — and remember the ops line: a serving stack is a production system with pagers attached. Capacity-plan in tokens/sec, not requests/sec; requests vary by two orders of magnitude in size, and Little’s Law works fine with tokens.