Broadcast, gossip and ordering guarantees
Between “send one message” and “agree on everything” sits a family of primitives systems quietly rely on.
Between “send one message” and “agree on everything” sits a family of primitives that real systems lean on constantly and name rarely. This chapter names them — because once you can say precisely what delivery guarantee a component needs, half of its design argument is over.
The guarantee ladder
| Broadcast | Every correct node gets… | Order promise |
|---|---|---|
| Best-effort | the message, maybe | none |
| Reliable | the message, definitely (retransmit/relay) | none |
| FIFO | each sender’s messages in that sender’s order | per sender |
| Causal | anything that happened-before a message, first | respects causality (Lamport, applied) |
| Total order | every message, in one agreed sequence | identical everywhere |
Each rung costs more coordination than the one below, so the skill is buying the cheapest sufficient rung. A metrics fan-out is happy with reliable. A chat room wants causal — the reply must not arrive before the question. Replicated state machines need total order — and here is the punchline this chapter exists for: total order broadcast and consensus are the same problem. If every node delivers the same messages in the same order, you have agreed on a sequence; that is exactly what Raft’s log does. The next chapter is this rung, taken seriously.
Gossip — the anti-broadcast
For data that tolerates lag — membership, load hints, configuration — flooding everyone reliably is overkill. Gossip spreads it the way rumours spread: each node periodically tells a few random peers what it knows; infected nodes infect others. Convergence is O(log n) rounds, no coordinator, no hot spot, and node failures barely dent it — which is why Dynamo runs membership on gossip and Cassandra still does. The price is the name: eventual, probabilistic, and briefly inconsistent — never put an invariant on it. Its quieter siblings do repair work: anti-entropy (periodically diff replicas — Merkle trees make the diff cheap — and heal), and read repair (notice divergence while answering a read, fix it in passing).
Where it bites
- Buying total order for telemetry — paying consensus prices where reliable would do.
- Assuming FIFO across senders: two producers’ messages interleave arbitrarily unless something orders them.
- Gossip as a source of truth: it is a weather report, not a ledger.
- Skipping the hands-on: the linked Gossip Glomers challenges turn this whole chapter from vocabulary into muscle memory in a weekend.