In production: feedback, drift and improvement loops
Offline evals said ship it; now the system meets users, and the measuring must continue.
The evals chapter ended with a suite gating your deploys; offline, the story is controlled. Then the feature meets users, and two inconvenient truths surface: your eval set is a photograph of last month’s traffic, and the model under you changes — your users drift, your provider retrains, your product grows a new audience. This chapter is the measuring that never stops.
Online measurement
Offline scores predict; only production confirms. Ship AI changes like the System Design delivery chapter taught — canary first, then an A/B test where the metric is a real outcome (task completed, ticket deflected, code merged), not a judge score. Around it, guardrail metrics that page when the canary misbehaves: error and refusal rates, latency percentiles, cost per request, escalation-to-human rate. And mine the feedback users already give without being asked: accepted the draft, edited it heavily, retried the request, abandoned the flow — implicit labels, free, honest, and at full production scale. Thumbs-up buttons are sparse and biased; the edit distance on your suggestion is neither.
The review queue, and drift
Random sampling finds failures at their base rate, which is slow. Route the suspicious slice to human review instead: low judge scores, user retries, heavy edits, very long or very short outputs, new-intent clusters. Weekly, an engineer reads a hundred of these transcripts — unglamorous, and the single highest-yield habit in applied AI, because every failure found becomes an eval case (the flywheel below). What the queue surfaces over time is drift in both its forms: your traffic shifting under a frozen eval set (new user segment, new jargon, new use case your prompts never met), and the model shifting under frozen traffic — a provider update, or your own “harmless” prompt edit. The defence is the same for both: refresh eval sets from live traffic continuously, and re-run the suite on a schedule, not only on your own deploys — the thing you depend on deploys too.
The flywheel, and the bad day
Production traffic → review queue → new eval cases → prompt/retrieval/routing fixes → gated deploy → better traffic. Teams that spin this loop weekly compound; teams that shipped and moved on decay quietly until a screenshot goes viral. For the bad day itself, have the System Design failure chapter pre-answered: a kill switch per AI feature (flag-controlled, tested), a designed fallback (smaller model, cached responses, the non-AI path — an AI feature without one is an availability decision someone made by accident), and postmortems for bad outputs, not just bad uptime — “the model told a customer X” deserves the same blameless rigour as an outage, and usually yields a better guardrail.
Where it bites
- An eval set frozen at launch — six months later it certifies a product nobody uses that way any more.
- Judge scores as the online KPI — the judge approves; the users churn. Outcomes outrank opinions, including the judge’s.
- Cost discovered on the invoice — tokens per feature per day is a dashboard, or it is a surprise.
- No kill switch: the bad output is live, trending, and your fastest mitigation is a deploy pipeline.