The classical toolbox

A decade into the deep learning era, most production models on tabular business data are still trees and linear models — because on structured data they train in seconds, explain themselves, and routinely win. Knowing this toolbox is not retro; it is what keeps you from spending GPU money on a problem a forest solves at lunch.

Trees, and the two ensembles that rule tables

A decision tree is learned if-else: repeatedly split the data on the feature/threshold that best purifies the classes. One tree overfits with enthusiasm — grow it deep enough and it memorises. The fix is crowds. Random forests: train hundreds of trees on random subsets of rows and features, average them — variance cancels, and the thing is nearly tuning-free. Gradient boosting (XGBoost, LightGBM): train small trees sequentially, each correcting the residual errors of the sum so far — usually the accuracy champion on tabular data, at the price of more knobs and less forgiveness. Rule of thumb the leaderboards keep confirming: tables → boosted trees; images, audio, language → deep learning (next chapter, where the features themselves must be learned).

Distance, dimensions, and a warning about embeddings

k-nearest neighbours barely counts as learning — store everything, predict from the k closest examples — but it teaches the geometry that retrieval will later depend on: in high dimensions, distances concentrate; everything becomes nearly equidistant, and “nearest” loses meaning. This curse of dimensionality is why raw high-dimensional vectors disappoint, and why learned embeddings (which pack meaning into a few hundred dense dimensions) work where raw features do not. When retrieval quality mystifies you in a later chapter, the explanation often lives here.

Structure without labels, and the regulariser dial

Unsupervised tools find shape in unlabelled data: k-means partitions into k clusters (you choose k, and it finds spheres whether or not spheres exist); PCA finds the directions of greatest variance — compression, visualisation, denoising in one move. On the supervised side, the dial you will turn forever is regularisation: penalise large weights (L2) to smooth the model, or penalise non-zero weights (L1) to force feature selection. It is the bias–variance tradeoff with a lever attached, and the first thing to reach for when validation lags training.

Feature engineering — where the domain enters

Classical models see only the columns you give them, so the craft is making columns that carry signal: encode categories (one-hot for few, target/embedding encodings for many — carefully, that one leaks), scale magnitudes for distance-based methods, build interactions the model family cannot invent (trees find thresholds, not ratios — hand them debt/income). This is where ten years of domain knowledge outperforms ten more layers, and it is the most transferable skill on this page: LLM context engineering, later, is feature engineering by another name.

Where it bites

• Reaching for deep learning on 5,000 rows of tabular data — the forest wins, and explains itself on the way.
• k-means on unscaled features — the loudest column decides the clusters alone.
• Feature importance read as causation — it ranks usefulness to this model, nothing more.
• Target encoding fitted on the full dataset — leakage, again, wearing a clever hat.