Breaking it on purpose

Everything in this pathway so far is a claim: committed entries survive, the lease fences, replicas converge. A distributed system you have not deliberately broken is a stack of such claims awaiting their first audit — scheduled, typically, by production, at night. The professional alternative is to run the audit yourself, and there are four escalating ways.

Jepsen-style testing — the empirical audit

The method behind those database autopsies in the replication chapter is simple to state: generate concurrent reads and writes from many clients; inject real faults — partitions, clock skew, process kills — mid-workload; record every operation’s invocation and result; then check the history against the claimed consistency model with a linearizability checker. The last step is the innovation: correctness as a property of the observed history, not of the marketing page. The recurring finding across a decade of analyses: systems keep their promises in calm seas and shed them during partitions and leader elections — precisely where you stopped watching.

Deterministic simulation — the time machine

The nastiest bugs need five improbable events in one order, so waiting for them is hopeless. Deterministic simulation testing runs the whole cluster in one process with simulated time, network and disks, driven by a seeded RNG: now the five-event catastrophe is just seed 8,412,113 — found by running millions of seeds overnight, and replayed exactly under a debugger. FoundationDB built its reputation on this, and TigerBeetle after it; the buy-in is real (all I/O behind swappable interfaces, no stray nondeterminism), which is why it is an architecture decision, not a test you add later.

Chaos engineering — the production audit

Staging cannot reproduce production’s topology, data or traffic, so the discipline in this chapter’s one-page reading runs experiments where the truth lives: state a hypothesis (“steady state survives one zone loss”), minimise blast radius (one instance, small traffic slice, business hours, hand on the abort switch), inject, observe, widen slowly. Chaos without a hypothesis is vandalism; with one, it is the cheapest failover rehearsal you will ever run. The fault menu worth rotating through: partitions (including asymmetric ones — A hears B, B not A), clock jumps, slow-not-dead disks and dependencies (gray failure beats clean death for damage), and the loss of exactly the coordination service the last chapter made you depend on.

Formal specification — the audit before the code

For the protocol at your system’s heart, a TLA+ specification — states, transitions, invariants — lets a model checker enumerate every reachable interleaving of a small instance and hand you the exact trace that violates an invariant. Amazon’s teams famously caught subtle bugs in shipped designs this way, some requiring thirty-plus steps to trigger — planning-poker odds of finding by testing: zero. Spec the consensus usage and the commit path; skip the CRUD.

Where it bites

• Testing only clean crashes — gray failure (slow, flaky, half-partitioned) is the production distribution.
• A failover “tested” once, at launch, two years and forty deploys ago — rehearsals expire.
• Chaos with no steady-state metric — you cannot see the experiment fail.
• Trusting the checker’s green on a three-node, five-op model to mean the 200-node deployment is safe — the model checks the protocol; operations remain yours.