Talking between machines

Before the deep theory, the plumbing: one service calling another. The industry’s recurring mistake here is thirty years old and still shipping — making a remote call look like a local function call. The 1994 paper in this chapter’s reading demolished the idea for good: a remote call can fail partially, takes ten million times longer, and can succeed invisibly. An API that hides those facts does not remove them; it removes your handling of them.

Serialisation and the evolution problem

Bytes on the wire need an agreed shape. JSON is self-describing, human-readable and verbose; binary formats like Protobuf and Avro are compact, fast, and schema-first. The schema is the real prize — not the bytes saved, but the evolution rules: numbered fields, unknown fields preserved, and a written definition of what a reader must tolerate. Because here is the deployment fact of life: you can never update every service at once, so old readers meet new messages and new readers meet old messages, every single deploy. The rules that keep that safe fit on a card: add fields as optional, never reuse a removed field’s number or name, never change a field’s type in place, and widen only in ways old code ignores gracefully. Breaking any of them is a distributed outage with a one-deploy fuse.

Contracts, discovery, connections

gRPC-style stacks package the good defaults: a schema-defined contract, streaming, and — the underrated one — deadlines that travel. A caller with 200 ms left tells the callee, which tells its callee, so work abandoned upstream is cancelled downstream instead of completing uselessly (remember the retry-storm spiral in System Design — deadline propagation is its vaccine). Finding the callee at all is service discovery: addresses change constantly under autoscaling, so callers resolve a logical name against a registry kept fresh by health checks — DNS at its simplest, a dedicated registry once churn is real. And between any two services, keep connection pools warm: the Foundations handshake tax, amortised.

Where it bites

• An interface “cleanup” that renumbers Protobuf fields — old readers now parse new messages as garbage, silently.
• Deadlines set only at the edge: inner services finish work nobody is waiting for, and the queue grows.
• Retrying a non-idempotent RPC because the client library made it one flag.
• Treating discovery staleness as impossible — callers holding dead addresses for minutes after a scale-down.