Language Models are Few-Shot Learners

The problem: a new training run for every task

By 2019 the recipe for language tasks was settled: pretrain a Transformer on a large corpus, then fine-tune a copy of it on thousands of labelled examples for each task you cared about. It worked well and it did not scale as a way of working. Every task needed a dataset, a training run and its own copy of the weights; the labelled data was often the bottleneck; and a model tuned for one task forgot how to do the others. People, by contrast, pick up a new task from a sentence of instruction and a couple of examples. GPT-2 had hinted that a big enough language model might too.

The idea: make the model large enough that the prompt is the program

The authors trained a decoder-only Transformer with 175 billion parameters, ten times larger than anything before it, on about 300 billion tokens of filtered web text, books and Wikipedia, and then never updated its weights again. Instead they evaluated it by in-context learning: writing the task into the prompt, either as an instruction alone (zero-shot), one worked example (one-shot), or a handful (few-shot, up to what fits in the 2,048-token context). The model continues the text, and continuing it correctly is the task.

How it works

The paper is mostly measurement. Eight models from 125 million to 175 billion parameters were trained the same way and tested on more than two dozen benchmarks in each of the three prompting modes, and the headline chart shows two things at once: accuracy rises steadily with size, and the benefit of adding examples to the prompt rises with size too. The small models barely use the examples; the large one reads them like instructions.

Some of the results surprised the authors. The largest model did two-digit addition nearly perfectly from a few examples, unscrambled words, translated between languages it had seen mostly incidentally, and wrote news articles that human readers identified as machine-written only slightly better than chance. Others were honest failures: common-sense reasoning benchmarks and some reading-comprehension tasks stayed well behind fine-tuned systems, and the paper devotes a section to the bias the model absorbed from the web and the energy its training consumed.

What it changed

GPT-3 turned a model into an interface. OpenAI exposed it through an API weeks after the paper, and “prompt engineering” became a job description; the instruction-following training that produced ChatGPT two years later was built on exactly this model and exactly this observation. The scaling thesis, that capability keeps arriving with size, set the research agenda and the capital expenditure of the field for the years that followed, and the next paper on this list is the measurement that justified it.

Where you meet it today

• Every chat and copilot product. A system prompt plus examples is few-shot prompting with a nicer interface.
• Prompt templates in code. The f-string with three examples in it is this paper, in production.
• Evaluation by prompting. Benchmarks now report zero- and few-shot numbers by default because this paper did.
• The debate about what in-context learning is, which is still open and still shapes how models are trained.