The machine underneath
You write code that works, but you cannot yet say what it costs.
Every program you will ever write runs on the same physical arrangement: a processor that executes simple instructions absurdly fast, and data that lives progressively further away from it. The processor is almost never the bottleneck. Waiting for data is. This stage builds the cost model that every later decision — cache or not, index or not, one service or two — silently consults.
The parts, in one paragraph
A modern server CPU has a handful to a few dozen cores, each executing billions of instructions per second on values held in registers. Between the cores and main memory sit two or three levels of cache — small, fast copies of recently used memory (L1 is per-core and tiny, ~32 KB; L3 is shared and a few tens of MB). Below that: RAM in the tens or hundreds of GB, then SSDs in the TBs, then the network, which is how you reach everything you do not have. Each step down is bigger, cheaper per byte, and slower — usually by one or two orders of magnitude.
The numbers worth memorising
These are the classic “latency numbers every programmer should know”, rounded to be memorable. The absolute values drift year to year; the ratios are what design decisions turn on, and they have been stable for decades.
| Operation | Cost | Intuition |
|---|---|---|
| L1 cache hit | ~1 ns | the baseline — call it one heartbeat |
| Branch mispredict | ~3 ns | the CPU guessed the if-branch wrong and rewinds |
| Mutex lock/unlock, uncontended | ~20 ns | cheap — contention is what costs, not the lock |
| Main memory reference | ~100 ns | 100× the cache — the first big cliff |
| Read 1 MB sequentially from RAM | ~10 µs | sequential is the fast lane everywhere |
| SSD random read | ~100 µs | 1,000× RAM — the second cliff |
| Read 1 MB sequentially from SSD | ~1 ms | still fine — again, sequential forgives |
| Round trip, same datacenter | ~0.5 ms | a network hop costs like an SSD read |
| Spinning disk seek | ~10 ms | mechanical — why databases fear random disk I/O |
| Round trip, cross-continent | ~150 ms | physics; no vendor sells a way around it |
Scale it to human time and the ladder stops being trivia: if a cache hit takes one second, main memory is a two-minute walk, an SSD read is a long weekend, and a request to another continent is a five-year expedition. One line of code that quietly crosses the network sits at the bottom of that ladder no matter how elegant it looks.
Cache lines and locality
Memory does not move one byte at a time. It moves in cache lines of 64 bytes: touch one byte and the hardware fetches its 63 neighbours, betting you will want them next. That bet is the whole story of fast code. Walk an array front to back and every fetch pays for the next several elements — the prefetcher even starts fetching ahead of you. Chase pointers through a linked list or a tree scattered across the heap and every hop is a fresh ~100 ns miss: same big-O, ten times slower.
The classic demonstration is iterating a large 2-D array. Loop row-by-row (the order it is laid out in memory) and you stream through cache lines; loop column-by-column and every single access lands on a different line. Identical arithmetic, routinely a 5–10× difference. When a profiler says your time is in “memory”, this is usually what it means.
Where memory comes from
Your language gives you two allocation regimes. The stack is a bump of a pointer — effectively free, automatically reclaimed, and beautifully local. The heap is where dynamic and long-lived things go, and it costs real work: the allocator finds space, and in garbage-collected languages every allocation is a future collection to pay for. Allocation inside a hot loop is one of the most common self-inflicted slowdowns in production code — not because one allocation is slow, but because a million of them per second turns the allocator and the GC into your busiest component. Reuse buffers; allocate outside the loop; let flat arrays of values beat trees of boxed objects.
Where it bites
- Pointer-heavy designs at scale. The elegant object graph benchmarks fine on 100 items and dies on 10 million — every edge is a probable cache miss.
- False sharing. Two threads updating two different counters that happen to sit in the same 64-byte line will fight over that line and serialise each other. Pad hot per-thread data apart.
- Benchmarking with a warm cache and shipping to a cold one — or the reverse. Say which case you measured.
- Trusting big-O across the ladder. O(log n) probes of an SSD lose to O(n) over a RAM-resident array for astonishing values of n.
If you want the full self-taught CS path, take this reading order rather than assembling your own.
The latency ladder as an interactive chart, year by year. Worth revisiting until the ratios are reflex.