ImageNet Classification with Deep Convolutional Neural Networks

The problem: computers could not see, and progress had stalled

In 2011 the best way to recognise objects in photographs was a pipeline designed by hand. Engineers wrote detectors for edges, corners and colour patches (SIFT, HOG and their relatives), pooled them into a fixed-length description of the image, and trained a classifier on top. It worked, slowly, and it had hit a ceiling: on ImageNet, a benchmark with 1.2 million photographs in a thousand categories, the error rate improved by a point or two a year. Neural networks that learned their own features had existed since the 1980s and were widely considered a dead end for anything but small problems, because they were slow to train and overfit badly.

The idea: a deep network, trained on the whole dataset, on graphics cards

Krizhevsky, Sutskever and Hinton did not invent the convolutional network; LeCun had read cheques with one in the 1990s. What they did was make one large enough to be worth it and find the handful of engineering decisions that let it train: eight learned layers and sixty million weights, fed by a dataset big enough to need them, on two consumer GPUs programmed to do the arithmetic quickly. The ingredients were individually known. The combination won the 2012 ImageNet competition with a top-5 error of 15.3 per cent against 26.2 per cent for the runner-up, a gap larger than the previous several years of progress combined.

How it works

A convolutional layer slides a small filter across the image and records how strongly each patch matches it; a layer has many filters, so it produces a stack of feature maps, and the next layer slides its filters across those. Early layers end up detecting edges and colour blobs, as the hand-designed features did, except that these were learned from the data and the later layers learn combinations of them that no engineer would have written. Pooling layers shrink the maps between stages so later filters see a wider field.

Four decisions made it trainable. The ReLU activation, max(0, x), replaced the smooth sigmoids of older networks and trained several times faster because its gradient does not vanish. Dropout randomly silenced half the neurons in the large fully connected layers during training, which forced redundancy and cut overfitting dramatically. Data augmentation, random crops and flips and colour shifts, multiplied the effective dataset by thousands. And the GPU implementation, written by Krizhevsky himself, made a week of training feasible instead of a year. Each of these is now a default setting; in 2012 each was a discovery.

What it changed

AlexNet is the start of the deep-learning era by common agreement. Within a year every serious entry to ImageNet was a deep convolutional network; within three, networks were deeper than a hundred layers and better than people at the benchmark. The techniques spread from vision to speech, then to language, and the demand for GPU compute that the paper kicked off reshaped an industry. The three authors’ later work (Sutskever at OpenAI, Hinton’s lab, Krizhevsky at Google) traces a direct line from this paper to the models of the 2020s.

Where you meet it today

• Every image model, from a phone’s face unlock to medical imaging, is a descendant; convolutions remain in use even where Transformers have taken the top layers.
• ReLU, dropout and augmentation are default options in every deep-learning framework.
• GPUs as the unit of ML compute. The paper is the reason NVIDIA is an AI company.
• Transfer learning. Reusing a network trained on ImageNet as the starting point for another task began with AlexNet’s features.