Anatomy of a request
Every system you will ever design is, at bottom, this one picture with more boxes.
Strip any architecture back far enough and one picture remains: a request crossing a short chain of boxes, spending time in each. System design is the discipline of choosing those boxes deliberately — so start by making the picture concrete, and by attaching the three numbers every design conversation runs on.
The boxes, and why each exists
DNS turns the name into an address (and, with region-aware answers, quietly does your first load balancing). A CDN serves static and cacheable content from the edge — it wins by removing distance, Foundations’ most expensive commodity. The load balancer spreads traffic across interchangeable app servers and survives the loss of any one of them; know the two kinds — L4 balances TCP connections (fast, content-blind), L7 reads HTTP and can route by path, cookie or header (richer, costlier), and it is usually where TLS terminates. Behind it, stateless app servers — stateless precisely so any server can take any request, which is what makes horizontal scaling and zero-drama deploys possible; state that must live somewhere goes to the cache and the database, which get their own stages.
The three numbers
Latency is time per request; throughput is requests per time; they are cousins, not synonyms — a system can be high-throughput and slow (batch pipelines) or low-throughput and fast. Availability is the fraction of requests that succeed, spoken in nines:
| Availability | Downtime / year | Downtime / day |
|---|---|---|
| 99% | ~3.7 days | ~14 min |
| 99.9% | ~8.8 hours | ~86 s |
| 99.99% | ~53 min | ~8.6 s |
| 99.999% | ~5.3 min | ~0.9 s |
Two compositions to internalise. Serial dependencies multiply: a request that must touch five 99.9% services is a ~99.5% request — chains eat nines, which is a standing argument for fewer required dependencies and graceful degradation for the rest. And tails dominate experience: design at the p99, not the average — in a page that fans out to twenty services, nearly every user touches somebody’s p99. One more relation quietly governs capacity everywhere: Little’s Law — concurrency = arrival rate × time in system. 1,000 req/s at 200 ms each means ~200 requests alive in your process at all times; that is your connection pools, memory and thread counts, derived in one multiplication.
The budget, worked
Give the request a number and make the hops fit. Say the product wants p99 ≤ 300 ms for a page in the user’s region: TLS + edge ~20 ms, LB + app ~5 ms, two cache reads ~2 ms, one DB query budgeted 30 ms, rendering 20 ms — roughly 80 ms spent, 220 ms of headroom that vanishes the moment someone adds a cross-region call (+150 ms) or a sequential fan-out. Budgets convert “make it fast” into decisions: cache this, co-locate that, parallelise the fan-out, delete the hop.
Where it bites
- Sticky sessions — state smuggled into “stateless” servers; one node’s death logs users out and defeats the balancer.
- Averages on dashboards, tails in tickets. Alert on percentiles.
- Serial fan-out that could be parallel: five 40 ms calls are 200 ms in series, 40 ms in parallel.
- No stated budget at all — then every hop is justifiable and the sum is not.