Consensus

Quorums let replicas converge, but some questions cannot tolerate “roughly agreed” — they need one answer, agreed, now, and still agreed after any minority of machines dies. Who is the leader? Did this transaction commit? Who holds the lock? Answer such a question twice, differently, and you get split brain and corrupted history. Getting a group of unreliable machines to answer it once is consensus.

Why the obvious approaches fail

“Have a coordinator ask everyone and commit if all say yes” is two-phase commit, and it has a fatal flaw as a consensus protocol: if the coordinator dies after collecting yes-votes but before announcing the outcome, every participant is stuck — voted, locked, and unable to decide alone. Blocking on a single failure is exactly what we were trying to escape. “Let the nodes vote by majority whenever they like” fails differently: without terms and careful rules, two candidates can each assemble overlapping support at different moments and both believe they won. Consensus protocols are precisely the bookkeeping that closes these gaps — and the arithmetic they all lean on is that any two majorities of the same group share at least one member.

Raft, in enough detail to trust it

Raft divides time into numbered terms. Every node is a follower, candidate, or leader. Followers expect periodic heartbeats from the leader; a follower that hears nothing for a randomised election timeout (the randomisation is what prevents endless split votes) increments the term and stands as a candidate, voting for itself and asking the rest. A node grants at most one vote per term, so two leaders in one term are arithmetically impossible: both would need majorities, and majorities intersect.

The leader’s job is a replicated log. It appends each client command, ships entries to followers (with the index and term of the preceding entry — followers reject on mismatch, which is how divergent histories get detected and repaired), and declares an entry committed once a majority holds it. Committed entries are then applied, in order, by every node’s state machine — the log is the agreement.

Two safety rules make crashes survivable. The election restriction: a node only votes for a candidate whose log is at least as up-to-date as its own — so a winner’s majority necessarily includes every committed entry, and a new leader can never erase committed history. And leaders never overwrite their own entries; they only repair followers toward their log. Result: a cluster of 2f + 1 nodes rides out f failures — five nodes, two losses, business as usual; lose the majority and the cluster stops accepting writes rather than lying. That stop is a feature.

The practical layer: leases and fencing

Real systems rarely put consensus on the data path — every committed write costs a majority round trip. Instead, a small consensus service (etcd, ZooKeeper, Consul) guards the control plane: configuration, membership, and leases — time-boxed leadership grants that must be renewed. One trap remains even then: a leaseholder that pauses (a long GC, say), loses its lease without noticing, and resumes acting on stale authority. The fix is the fencing token: every lease carries a monotonically increasing number, downstream systems remember the highest they have seen, and actions bearing an older token are rejected. Authority you cannot fence is authority you do not have.

Deployment notes worth knowing

• Run 3 or 5 voters — even counts add cost without adding failure tolerance; 5 tolerates 2 losses.
• Keep voters close: commit latency ≈ one round trip to the median voter. Cross-region consensus is a deliberate, priced decision.
• Throughput comes from batching and pipelining many commands per round trip — the protocol is not the bottleneck when operated well.
• Reads are subtle: serving them from the leader without a check risks stale reads from a deposed leader; production systems use lease-based reads or a quorum check.