Partial failure

On a single machine, failure is refreshingly honest: the process is running or it has crashed, and the operating system knows which. Split the work across two machines and a third state appears that has no equivalent in single-machine life — unknown. You sent a request. Nothing came back. Now what do you actually know?

Almost nothing, and that is the founding problem of the whole field. The request may have been lost before arriving. The server may have crashed after doing the work. The reply may have vanished on the way back. From where you sit these are indistinguishable, so a timeout is not information about what happened — it is a decision you make in the absence of information.

Naming the ways things fail

It helps to know which failure model you are designing for, because they cost wildly different amounts to handle:

Ordinary backend engineering lives in crash-recovery plus partitions. Keep the eight classic “fallacies of distributed computing” pinned somewhere — the network is reliable, latency is zero, bandwidth is infinite, the network is secure, topology doesn’t change, there is one administrator, transport cost is zero, the network is homogeneous — every one of them is an outage report waiting for a date.

Retries, done properly

The timeout decision has teeth. If you retry and the original request actually succeeded, the work happens twice — a payment, an email, a row. So retries force a choice that shapes every serious API: at-least-once delivery with idempotent operations, or at-most-once and accept loss. There is no exactly-once wire; “exactly-once” in marketing means “at-least-once, deduplicated”.

Idempotency is implemented, not declared. The standard mechanics: the client generates a unique idempotency key per logical operation and sends it on every attempt; the server records the key with the outcome (in the same transaction as the work — that detail is the whole trick) and, on seeing a repeat, returns the recorded outcome instead of re-executing. Keys need an expiry policy, and the record must survive crashes — which is why it lives in the database and not a process map.

And retry politely, because naive retries are how one slow dependency becomes a self-inflicted flood: cap attempts (two or three, not “until it works”), back off exponentially (1s, 2s, 4s…), and add jitter — randomise each delay — so a thousand clients that failed together do not return in lockstep and knock the service down again on schedule. Retry only idempotent operations, and only errors that can plausibly heal (a timeout, a 503 — not a 400).

Why bother, then

Every arrow in your architecture diagram is now a place where the answer can be “unknown”, and the machines on either side of it fail independently — one rack loses power, the rest keep serving. That independence is the raw material of fault tolerance and the entire reason distribution is worth its pain. The rest of this pathway is about paying for it honestly: ordering events without shared time (Stage 2), keeping copies that agree usefully (Stage 3), and getting a group to commit to one answer (Stage 4).