Shipping safely: deploys, flags and migrations

The failure chapter dealt with the outages that arrive; this one deals with the outages we ship. Most production incidents walk in through the front door wearing a deploy badge — which is excellent news, because unlike hardware and networks, this failure source is entirely under your process control.

Bounded blast radius as a deployment property

Blue-green: two identical environments; deploy to the idle one, flip traffic, keep the old one warm as an instant rollback. Clean, doubled infrastructure, and the flip is all-or-nothing. Canary: route 1%, then 5%, then 25% to the new version, watching error rates, latency percentiles and business metrics at each step — blast radius is the dial itself, and the honest modern default. The step that separates adults from tourists: automated rollback wired to the SLO — if the canary burns error budget, the system reverts without waiting for a human to notice the graph. Which implies the real prerequisite: rollback is a designed path, tested like the deploy, not a git-revert improvised at 3 a.m.

N and N+1 always run together

During any rollout, old and new code serve simultaneously — same traffic, same database, same queues. That single fact generates the compatibility law: every deploy must be safe alongside its predecessor. Message and API changes follow the RPC chapter’s rules; the database gets the famous three-step, expand–migrate–contract: (1) expand — add the nullable column/new table, code reads old, writes both; (2) migrate — backfill in rate-limited batches (never one giant UPDATE holding locks on a hot table — the writes chapter explains exactly why); (3) contract — once nothing reads the old shape, drop it, in a later deploy. Slower than the big-bang rename? Yes. It is also the difference between a schema change and an outage with a schema change inside it. Renames are the classic trap: a rename is an add + a drop, never a rename.

Flags: decoupling deploy from release

The sharpest tool in the drawer: ship code dark behind a feature flag, then release by flipping a percentage — per user, per tenant, per region — and unrelease in seconds without a deploy. This is also the kill switch the AI chapters demanded, and how deploy risk (did the binary break?) gets separated from release risk (does the feature behave?). The taxonomy in Hodgson’s essay is worth internalising because the lifetimes differ: release flags die in weeks (delete them — flag debt is real code debt: every stale flag doubles the test matrix), ops flags (kill switches, load-shedding levers) live forever and get tested like circuit breakers, permission flags are entitlements wearing a flag’s clothes and belong in the auth system eventually.

Where it bites

• The Friday migration that renames a column — v(N) is still reading it; instant 500s fleet-wide.
• Canary judged on error rate alone — latency p99 and the business metric are how quiet degradations ship.
• Rollback that “mostly works” — if the new code wrote data the old code cannot read, your rollback is forward-only; expand–contract exists for this too.
• Two hundred live flags, ownerless — nobody can say what production even is. Flags get owners and expiry dates at creation.