The History of AI

Scaling laws

Era 4 · Deep learning and foundation models (2012–2021) › Transformers and language models

Lineage
Neurons and networks
Kind
Publication

Who

OpenAI, Johns Hopkins University

What happened

Jared Kaplan of Johns Hopkins University and OpenAI, with nine OpenAI researchers, posts "Scaling Laws for Neural Language Models" to arXiv on January 23, 2020. The paper measures how a language model's error changes as its size, data, and compute vary.

Why it mattered

Error falls as a power law in each of the three, predictably, across several orders of magnitude. That makes it possible to work out in advance how well a bigger model will perform.

The full story

Choosing a model's size, how much data to feed it, and how long to train it had been largely a matter of trial and error. Kaplan's team trained a large number of Transformer language models of different sizes and measured their loss, the error they make when predicting text.

What came out was a simple rule: loss falls as a power law with model size, with the amount of data, and with training compute, and some of the trends hold across more than seven orders of magnitude. Other details, such as how wide or deep the network is, barely matter within a broad range. Simple equations also describe when overfitting sets in and how quickly a model trains.

That leads to practical advice. Large models make much better use of each piece of data, so for a fixed compute budget, the efficient choice is to train a very large model on a relatively modest amount of data and stop well before it converges.

Four months later, the GPT-3 paper leaned on these laws to justify its size: loss should keep improving smoothly as the model grew. In 2022, DeepMind's Chinchilla revised the balance between parameters and data that Kaplan had proposed.

Under the hood

The laws take the form L(N) ≈ (N_c / N)^α_N, where L is the loss and N is the number of parameters, not counting the word embeddings. Data D and compute C behave the same way, each with its own exponent: about 0.076 for parameters, 0.095 for data, and 0.050 for compute.

An exponent that small means diminishing returns. With α_N = 0.076, halving the loss would take roughly 9,000 times as many parameters. The rule holds as long as none of the three factors is holding back the others.

Common misconceptions

  • What people often say: Scaling laws say all you need is bigger models.

    In fact: They describe the error in predicting text, not specific abilities, and only while parameters, data, and compute grow together. In 2022, Chinchilla showed it paid to use more data than Kaplan had proposed.

Leads to

  • Chinchilla (March 29, 2022)

    Documented direct influence

    Chinchilla contradicts and corrects the ratio Kaplan proposed

  • GPT-3 (May 28, 2020)

    Documented direct influence

    The GPT-3 paper justifies its size with scaling laws, which predict that validation loss improves as a smooth power law in model size (sections 1 and 2.1)

  • Anthropic is founded (January 2021)

    Documented direct influence

    The announcement of Anthropic's first funding round presents its team as the authors of earlier research on GPT-3, scaling laws, and learning from human preferences

Threads that pass through here

Sources

Original source

How to cite

“Scaling laws.” The History of AI, revised November 13, 2026. https://thehistoryofai.com/en/milestone/scaling-laws

This entry was last reviewed on November 13, 2026.

Open in the interactive timeline