Deep learning fundamentals

Classical models plateau where features are the hard part — nobody can hand-write the columns that describe a cat photo. Deep learning’s bargain: stack simple layers and let training discover the features itself, layer by layer. This chapter is the machinery of that bargain — build it once small, and the trillion-parameter versions stop being magic.

The network, and the loop that trains it

A multilayer perceptron is alternating steps: multiply by a weight matrix (Foundations’ maths chapter, cashed in), then apply a simple nonlinearity like ReLU. The nonlinearity is load-bearing — without it, ten stacked linear layers collapse into one. Training is a four-beat loop run millions of times: forward pass (predict), loss (score it), backward pass — the chain rule walked backwards through the stack, yielding every weight’s blame for the error — and update: nudge each weight downhill by a step scaled by the learning rate. Minibatches make the gradient affordable; Adam gives each weight an adaptive step; warmup-then-decay schedules tame the start and polish the finish. Karpathy’s course in the reading builds exactly this by hand — the single highest-leverage exercise in the pathway.

Why deep was hard, and the four fixes

Depth multiplies gradients layer by layer, so they vanish (early layers learn nothing) or explode (training detonates). Four standard parts fixed this, and every modern architecture — transformers included — is assembled from them: careful initialisation (start weights at scales that preserve signal), normalisation layers (keep activations well-behaved between layers), residual connections (let each layer learn an adjustment to a passed-through identity, so gradients have a highway), and dropout (randomly silence units during training so no neuron becomes indispensable — regularisation, network edition). Reading loss curves is the daily craft: train falling with validation flat is overfitting (more data, more dropout); both flat is underfitting or a learning rate too timid; loss spiking is a rate too bold.

Embeddings — the bridge to everything after

Networks eat vectors, so discrete things — words, users, products — get a learned vector each: an embedding. Training arranges the space so that things used similarly end up near each other; geometry becomes meaning. This one idea powers retrieval (nearest neighbours in embedding space), recommendations (users near items), and the first layer of every language model. When the retrieval chapter measures “semantic similarity”, it is measuring distances in exactly this kind of space — with the curse-of-dimensionality caveats the previous chapter filed.

Architectures, and why attention ate them

Architecture is prior knowledge baked into wiring. CNNs hardwire “nearby pixels relate, patterns repeat anywhere” — convolutions scan a shared filter across the image. RNNs hardwire sequence — consume tokens one at a time, carrying a memory — and paid for it twice: long-range memory decayed, and the one-at-a-time loop defied parallel hardware. Attention removed both constraints — every position looks directly at every other, all computed as one big matrix multiply — which is precisely the shape GPUs love (Foundations: sequential beats scattered; batch beats chatty). The transformer won as much on hardware fit as on cleverness, and that story continues in the next chapter.

Where it bites

• Judging a run by training loss — validation is the only loss that counts.
• Skipping the baseline: a linear probe or boosted trees first, or you cannot price what the network bought you.
• Learning rate treated as a detail — it is the single most consequential knob; sweep it first.
• Data starvation: deep models are data-hungry, which is why the practical entry point is fine-tuning something pretrained (a later chapter), not training from scratch.