Membership, failure detection and coordination

Consensus gave you a kernel of perfect agreement. This chapter is the machinery that rations it: deciding who is alive, who leads, and who holds which lock — using the expensive kernel as rarely as possible.

Deciding who is alive (you can’t, so decide anyway)

Chapter one proved a slow node and a dead node are indistinguishable from outside, so every “failure detector” is really a suspicion policy. Simple heartbeats with a fixed timeout misfire in both directions — too short flags GC pauses as deaths, too long leaves ghosts serving traffic. Better detectors adapt: phi-accrual tracks the arrival-time distribution and emits a continuous suspicion level, so the threshold tunes itself to network reality. At cluster scale, everyone-pings-everyone melts; SWIM (this chapter’s paper) fixes it with two tricks — ping a random member each round, and on silence ask k others to ping the suspect for you (routing around a bad link), then piggyback the verdicts on the pings themselves. Membership costs O(1) per node per round, and the design carries Consul and friends today.

The coordination service pattern

Rather than every application implementing Raft, one small replicated service — ZooKeeper, etcd, Consul — exposes the agreement kernel as a tiny filesystem with versions and watches. The classic recipes are one page each: a lock is an ephemeral node that vanishes if your session dies (no orphaned locks); an election is “lowest sequence number wins, watch the one ahead of you” (no thundering herd); config is a watched key every instance reacts to. The ZooKeeper paper’s lasting insight is the split: the service orders only the small coordination writes; the bulk data path never touches it.

Leases and fencing, revisited where they live

Every grant of authority from such a service is a lease — leadership for 10 seconds, renewable — because a grant without expiry plus a dead grantee equals a stuck system. And every lease needs the fencing token from the consensus chapter: the pause-and-resume zombie leader is not a hypothetical, it is the canonical outage of this architecture. Downstream systems reject stale tokens; authority you cannot fence is authority you do not have. Say it in reviews until it is boring.

Where it bites

• One timeout constant for “dead” across LAN, WAN and a GC-heavy runtime — adaptive detectors exist because this fails.
• Coordination service on the data path — it orders metadata; route bulk traffic elsewhere or watch it melt.
• Locks without sessions/ephemerality: the crashed client that owns a lock forever.
• Deleting the coordination you could have designed out — a partitioned keyspace where each node owns its keys needs no distributed locks at all; the best chapter here is the one you didn’t need.