Designing for failure

Everything so far designed the success path. Professionals are distinguished by the other one — because at scale, failure is not an event, it is a rate. Some request is timing out right now. The design question is never “will it fail?” but “what happens next, and how far does it spread?”

Define working, in numbers

An SLI is the measurement (fraction of requests under 300 ms and successful). The SLO is your target for it (99.9% over 30 days) — set from what users notice, not what the dashboard flatters. An SLA is the contractual version with penalties; never promise externally what you only aspire to internally. The complement of the SLO is the error budget: 99.9% over 30 days grants ~43 minutes of failure. Budgets turn reliability into an economy — burning fast pauses risky launches; running far under budget means you are paying for caution users cannot perceive. Alert on burn rate (how fast the budget is draining, on fast and slow windows), not on every blip: pages should mean “a human must act now”, or they train humans to ignore pages.

How failure travels

The worst outages are small failures amplified by the system’s own defences. The canonical spiral: a dependency slows; callers time out and retry, tripling load on a service already drowning; its queue grows until every request waits longer than its caller’s timeout — now all work is completed and discarded; the load balancer routes away from “unhealthy” nodes, concentrating the flood on survivors. Every defence made it worse. The counter-tools are all ways of refusing early:

• Timeout budgets that shrink down the call chain — if the edge promises 300 ms, an inner call may have 50; an inner timeout longer than the outer one is a bug you can grep for.
• Retries: capped, exponential, jittered, idempotent-only — and budgeted (e.g. retries ≤ 10% of traffic) so recovery cannot become the second outage.
• Circuit breakers — stop calling what keeps failing, probe gently, recover automatically:

• Bounded queues + load shedding: a fast “no” (429, degraded page) protects the “yes” for everyone else; an unbounded queue is a promise to fail slowly and completely.
• Backpressure: slowness propagates as reduced intake, not as infinite buffering — consumer lag on the Stage 5 log is backpressure made visible.
• Bulkheads: separate pools per dependency or tenant, so one storm cannot sink the ship; plus health checks that distinguish liveness (restart me) from readiness (route around me, briefly).
• Designed degradation: the page without recommendations, the search over yesterday’s index — built and tested in the calm, because improvised degradation at 3 a.m. is just a second incident.

Seeing and learning

Observability is three signals with different jobs — metrics (cheap, aggregated: rate, errors, duration percentiles per endpoint and dependency), logs (searchable detail, structured, stamped with a request ID), and traces (one request’s journey across services — the only way to see where the 300 ms went). Propagating a correlation ID everywhere is the cheapest observability investment you will ever make. And when the budget burns anyway: a blameless postmortem aimed at the system that made the mistake easy — the linked eighteen observations will reframe every incident review you attend — plus a periodic game-day where you kill things on purpose, because an untested failover is a rumour, not a capability.