Multi-region and disaster recovery
One region is a single point of failure with excellent marketing; going beyond it is a design, not a checkbox.
One region is a single point of failure with excellent marketing. But “go multi-region” is not a checkbox — it is a family of postures with order-of-magnitude cost differences, and the honest conversation starts with two numbers, not with architecture.
RPO and RTO — the two numbers
RPO (recovery point objective): how much data may be lost — the gap between the last safe copy and the disaster. RTO (recovery time objective): how long until service returns. “Zero and zero” is a wish, not a requirement; every posture below is a price on this pair, and the business — not the architect — owns choosing it. Ask per system: the payments ledger and the recommendation cache do not deserve the same numbers.
| Posture | RPO / RTO | What it really is |
|---|---|---|
| Backups + restore | hours / hours-days | the floor. Tested restores or it is a rumour, not a posture |
| Pilot light | minutes / ~an hour | data replicated cross-region; infra as code, ignited on demand |
| Warm standby | seconds-minutes / minutes | scaled-down full copy always running; scale up + fail over |
| Active-active | ~0 / ~0 | all regions serve; the hard problems of this pathway, at full price |
Active-active, honestly
Serving writes from two regions is the Distributed Systems pathway with real latency: cross-region round trips are ~60–150 ms, so synchronous replication taxes every write and asynchronous replication means a failover loses acknowledged writes (there is your non-zero RPO) — while accepting writes both sides is multi-leader, and “which region’s update to the cart wins” is the conflict chapter you already read. The professional pattern that dodges most of it: home the data — partition users/tenants to a primary region each (often legally required anyway: residency), serve them locally, replicate asynchronously for disaster, and reserve true multi-writer conflict resolution (CRDTs, LWW-with-eyes-open) for the few datasets that genuinely need it. Many “active-active” systems are honestly “active-active for reads, homed for writes” — and better for admitting it.
Static stability, and the failover you can trust
The Builders’ Library idea in this chapter’s reading deserves its name in your vocabulary: static stability — surviving a zone or region loss without any control-plane action. Pre-provision N+1 capacity so the survivors absorb the load without an autoscaler (which is down too, or panicking) having to act during the incident. The failover mechanics themselves: DNS-based (TTL-bound, minutes, clients that ignore TTLs exist) or anycast/global-LB (seconds); health checks that trip on region-level signals, with a manual override, because flapping between regions is worse than either region. And the rules that decide whether any of it works on the day: rehearse — an untested failover is a rumour (game days, the DS testing chapter, applied); beware the fail-back, which is often riskier than the failover (the returned region is cold-cached and stale — warm it deliberately); and drain toward your capacity, not toward hope.