Learning from data
Before any neural network: what it means for a machine to learn, and the discipline that keeps you from fooling yourself.
Machine learning is a single idea wearing many costumes: instead of writing the rules, show the machine examples and let it fit the rules. Everything in this pathway — up to and including the largest language model — is that idea at different scales. This chapter is the idea, plus the discipline that keeps you from fooling yourself with it.
The frame
Supervised learning in one sentence: given examples of inputs (features) and correct outputs (labels), find a function that maps one to the other, by defining a loss — a number scoring how wrong the current function is — and adjusting the function to shrink it. Predicting a number is regression; predicting a category is classification. The humble linear model — output as a weighted sum of features — remains the workhorse: fast, interpretable, and on tabular business data, embarrassingly hard to beat.
The only thing that matters: generalisation
A model that memorises its training data scores perfectly on it and uselessly on Tuesday. What you care about is performance on data the model has never seen— so the first discipline of the field is splitting: train on one slice, tune decisions on a validation slice, and hold out a test slice you touch once, at the end. Touch it twice and it has quietly become validation, and its score a press release. The failure modes have names: overfitting (great on train, poor on validation — the model learned the noise) and underfitting (poor on both — the model is too simple). The bias–variance tradeoff is the dial between them, and a learning curve — error vs training size — tells you which side you are on, and whether more data will even help.
Leakage — the silent killer
The most expensive bug in applied ML is not a bad model; it is a great score built on information the model will not have at prediction time. The classics: a “predict churn” feature that encodes the cancellation itself; scaling computed over the whole dataset before splitting (the test set leaked its statistics into training); random splits of time-series data (the model trains on the future). The tell is a score too good for the problem. The defence is one question asked of every feature: will this value exist, with this meaning, at the moment of prediction?
Metrics that match the problem
| Metric | Answers | Reach for it when |
|---|---|---|
| Accuracy | share of correct predictions | classes are balanced and errors cost the same — i.e. rarely |
| Precision | of the flagged, how many were real? | false alarms are expensive (spam filter eating real mail) |
| Recall | of the real, how many did we catch? | misses are expensive (fraud, disease screening) |
| F1 | harmonic balance of the two | you must trade them and want one number |
| ROC-AUC | ranking quality across all thresholds | comparing models before choosing an operating point |
| Calibration | does “70% confident” happen 70% of the time? | the probability itself feeds a decision |
The 99%-accurate fraud model that flags nothing (fraud is 1% of traffic) is the canonical trap: with imbalanced classes, accuracy is a compliment the majority class pays itself. Choose the metric from the cost of each error type — that choice is a product decision wearing a maths costume.