Evaluation and production
The demo works. This stage is the difference between a demo and a system.
Here is the property that separates this field from the rest of the curriculum: the system is non-deterministic, and its failures are fluent. A wrong answer arrives in confident prose, not a stack trace. Without measurement, development degenerates into vibes — tweak a prompt, admire three examples, and silently regress five things you did not try. Evaluation is the discipline that replaces vibes with numbers; production is where the numbers earn rent.
Building the eval suite
An eval is a set of real cases with a defined check, run on every change — unit tests for behaviour. Source the cases from reality: genuine inputs from traffic, the ugly ones from your failure reports, plus deliberately hard edges (ambiguous asks, adversarial phrasing, questions whose correct answer is “I don’t know”). Fifty good cases beat five hundred synthetic ones. Then check at the right strength, preferring the cheapest check that catches the failure:
| Check | Examples | Trust |
|---|---|---|
| Programmatic | JSON parses, code runs, cited doc exists, number matches source | highest — use everywhere it applies |
| Reference-based | matches or contains a gold answer | high, where a gold answer exists |
| Rubric + LLM judge | “is the answer grounded in the sources? complete? on-tone?” | useful at scale — but calibrate first |
| Human review | samples, disagreements, high-stakes lanes | the anchor the others are calibrated against |
The judge deserves its own caution: it is a model too, with biases (toward verbosity, toward the first option, toward its own phrasing). Before trusting it, score a sample by hand and check agreement; give it a rubric and a reference, not an open “is this good?”; and prefer pairwise comparisons (“which is better, A or B?” with order swapped) over absolute scores — models are more reliable comparators than graders. An uncalibrated judge is your own bias, automated.
Wiring it into the lifecycle
Prompts, retrieval settings, tool descriptions and model choices are code: version them, review them, and gate their deployment on the suite. The payoff compounds — a provider updates a model, or you want to try the cheaper tier, and instead of a week of anxiety you run the suite and read a diff. Model upgrades become deploys. Online, close the loop the way Stage 6 of System Design taught: canary the change, watch guardrail metrics (task success, escalation rate, cost per request, latency), and sample production traffic back into the eval set continuously — user behaviour drifts, and yesterday’s suite slowly stops representing today’s traffic. Traffic feeds evals; evals gate changes; changes meet traffic.
Production economics
- Latency: stream always (Stage 1 — perceived latency is time-to-first-token); cut output length before you cut anything else — decode time is linear in it; parallelise independent calls.
- Cost: route by difficulty — a small model for the easy 80% and a large one for the hard 20% beats one large model for everything; cache aggressively (identical and near-identical calls are common); keep stable prompt prefixes for provider-side caching; set token budgets per feature so cost is a design property, not a surprise.
- Reliability: provider outages and rate limits are ordinary dependencies — timeouts, budgeted retries, a fallback model, and a designed degraded mode, exactly as System Design Stage 6 prescribed. An AI feature without a non-AI fallback is an availability decision someone should have made on purpose.
That closes the four pathways. Everything here compounds with the others: the ladder priced the model call, distributed systems made your tool calls safe to retry, system design gave the loop its budgets and its telemetry, and evals gave the whole thing a definition of “working”. Professional is exactly that — not knowing one layer, but moving between them without changing gears.