Load balancing, gateways and discovery
The boxes between the internet and your code deserve to be understood, not inherited.
Between the internet and your code stand boxes most engineers inherit rather than understand — until the day the “simple” load balancer is the outage. This chapter opens the boxes: what balancing actually decides, what a gateway is for, and how traffic finds services that will not sit still.
L4 vs L7 — what each can see
An L4 balancer works at the connection layer: it sees IPs and ports, picks a backend per connection, and forwards bytes it never reads — nearly wire-speed, protocol-blind. An L7 balancer terminates TLS and reads HTTP: it balances per request, routes by path/header/cookie, retries idempotent failures, and emits the metrics you actually alert on. The costs mirror the powers: L7 pays parsing and holds state; L4 cannot help you with anything it cannot see — including one long-lived HTTP/2 connection carrying a thousand requests/second that L4 happily “balanced” onto a single backend. Real edges layer them: L4 (often anycast) at the front for scale, L7 behind it for brains.
Choosing a backend, and ejecting one
Round robin assumes requests cost the same; they do not. Least outstanding requests is the sturdy default — send work where the queue is shortest, which automatically feeds slow nodes less. Session affinity (consistent-hash on a cookie/user) buys cache locality at the price of the stateless ideal from the request-anatomy chapter — take it knowingly or not at all. Health checking is where balancers turn dangerous: a naive checker that ejects any instance failing a ping will, when a shared dependency hiccups, eject the entire fleet and route 100% of traffic to the last “healthy” victim — the retry spiral’s cousin. The counter-pattern is outlier ejection with a floor: eject the statistically worst few, never below a minimum healthy fraction — and distinguish liveness (“restart me”) from readiness (“route around me briefly”), the failure chapter’s split, enforced here.
Gateways and discovery
An API gateway is an L7 balancer with the cross-cutting jobs moved in: authentication, rate limiting, request logging, routing to the right service — excellent, precisely because every request now flows through one very smart single point of failure. Keep it boring: policy yes, business logic no; anything clever in the gateway is logic you cannot deploy independently and an outage you cannot scope. Behind it, service discovery answers “where is checkout-service right now” — a registry fed by health checks, consumed either server-side (the balancer resolves; simple clients) or client-side (callers subscribe and balance themselves; smarter, and the model service meshes automate by parking a proxy sidecar next to every instance — the mesh is this chapter productised, priced in operational complexity).
Where it bites
- gRPC/HTTP2 behind an L4 balancer — one connection, one backend, “balanced”.
- Health checks that test a shared dependency — one Redis blip ejects the fleet.
- Gateway timeout longer than the client’s — work completing for callers already gone (deadline budgets, again).
- Config in the balancer diverging from reality in the registry — the 3 a.m. classic; make one own the truth.