Scaling Laws for Neural Language Models

The problem: should we build a bigger one?

By the end of 2019 bigger language models had been better at every step from GPT to GPT-2, but nobody knew whether that would continue, by how much, or at what cost. Training the next model was a bet of millions of dollars on an unmeasured trend. Three questions had no quantitative answer. If we double the parameters, how much does the loss fall? If we have a fixed compute budget, should it go to a larger model or to more data? And does the shape of the network, its depth and width, matter as much as its size?

The idea: measure the trend, and it turns out to be a power law

Kaplan and colleagues at OpenAI trained hundreds of models across seven orders of magnitude of size and plotted the test loss against parameters, dataset size and compute. On logarithmic axes the points fell on straight lines. Loss as a function of model size, N, is (N_c/N)^0.076; as a function of data, D, it is (D_c/D)^0.095; as a function of compute, a similar law with exponent 0.050. The architectural details that researchers spent their time on, depth versus width, the number of heads, mattered far less than the sheer count of parameters, within wide limits.

How it works

The second finding is about allocation. Large models are more sample-efficient: they reach a given loss having seen fewer tokens. So for a fixed budget the paper concluded that most of the increase should go to model size, with data growing more slowly, and that a model should be trained well short of convergence. The calculator below uses the paper’s fitted constants, and lets you vary the data-to-parameter ratio, because the ratio is where the story continued.

In 2022, DeepMind’s Chinchilla paper re-ran the experiment with the learning-rate schedule set correctly for each run and found a different split: parameters and data should grow together, about twenty tokens per parameter. Under the original recipe Gopher had 280 billion parameters trained on 300 billion tokens; under the new one, the same compute bought a 70 billion-parameter model trained on 1.4 trillion tokens, and the smaller model won on almost every benchmark while costing a quarter as much to run. The power laws survived; the exponents and the advice did not.

What it changed

This paper, with GPT-3 six months later, is the quantitative case for scaling, and the reason training budgets went from millions to billions: the lines said the money would buy something. Chinchilla corrected the split; LLaMA and its successors then went past it, training small models on far more than twenty tokens per parameter because a model that is cheap to serve is worth more than one that is optimal to train. Scaling laws are now measured for everything, from data mixtures to inference-time compute, and “does it follow a clean scaling law” is the first question asked of any new technique.

Where you meet it today

• Model cards and launch posts quoting parameters and training tokens: the two axes of this paper.
• Compute-optimal versus inference-optimal as a design decision, when a team picks a model size.
• Loss curves on log axes in every training report, and the expectation that they be straight.
• The economics of the field. A small exponent means each halving of loss costs orders of magnitude more compute; that arithmetic is the reason for the data centres.