Measuring before believing
Estimates start the argument; profilers end it.
The cost chapter taught you to estimate; this one teaches you to check. The two skills are a pair: an estimate without measurement hardens into folklore, and a measurement without an estimate has no way to notice it is absurd. The uncomfortable founding fact of the field: intuition about where time goes is reliably wrong — even for experts, even about their own code. Nobody is exempt, which is why the profiler exists.
A method, before tools
When a system is slow, resist the urge to read code. Interrogate resources instead. The USE method asks three questions of every resource — CPU, memory, disk, network, connection pools, thread pools: what is its Utilisation (how busy), its Saturation (how much work is queued waiting), and its Errors. Saturation is the one intuition misses: a disk at 60% utilisation with a deep queue is a worse citizen than one at 90% with none — queueing is where latency hides. Ten minutes of USE across the resource list usually points at the guilty subsystem before any code is opened.
Reading a profile
A CPU profiler samples the call stack hundreds of times a second; the aggregate says where time truly went. The standard picture is the flame graph: stacks stacked, width = share of samples. Two habits make it useful. Read widths, not heights — a tall thin tower is deep but cheap, a short wide plateau is the money. And distinguish on-CPU profiles (computing) from off-CPU time (waiting on locks, disks, networks): a service at 5% CPU that is dog slow is waiting, not working, and needs the off-CPU story — which is usually told by traces (the System Design observability chapter) rather than the CPU profiler.
Benchmarking without lying to yourself
| Sin | Why it lies | Instead |
|---|---|---|
| One run, one number | variance eats the signal — the same code varies run to run | many runs; report median and p95, never the best |
| Cold vs warm confusion | first runs measure caches filling and JITs compiling | warm up, then measure; say which regime you report |
| Averages | one 2-second outlier hides in a nice mean | percentiles — the tail is the user experience |
| The idle-lab benchmark | production has neighbours, contention, real data sizes | load-test against realistic data and concurrency |
| Microbenchmarking a fiction | the compiler deletes code with unused results | consume outputs; prefer measuring the real path end to end |
And the discipline that outranks all technique: change one thing at a time, keep the numbers, and compare like with like. A performance claim without its conditions attached — dataset size, concurrency, warm or cold — is an anecdote wearing a lab coat.
A worked session
“The endpoint is slow.” First: slow how — p50 or p99, since when, correlated with a deploy? Dashboard says p99 tripled at yesterday’s release. USE pass: CPU fine, DB connection pool saturated — queue depth climbing. Off-CPU story: requests waiting for connections. Why now? The deploy added one query per item in a list — the N+1 pattern from the cost chapter, invisible on the five-item test fixture, ruinous on the two-hundred-item production account. Fix the query shape, watch the pool drain, keep the graph in the postmortem. Notice what did the work: a method, a queue depth, and one profile — not a hunch.