Scaling laws
Era 4 · Deep learning and foundation models (2012–2021) › Transformers and language models
- Lineage
- Neurons and networks
- Kind
- Publication
Who
- Jared Kaplan
- Sam McCandlish
- Tom Henighan
- Tom B. Brown
- Benjamin Chess
- Rewon Child
- Scott Gray
- Alec Radford
- Jeffrey Wu
- Dario Amodei
OpenAI, Johns Hopkins University
What happened
Jared Kaplan of Johns Hopkins University and OpenAI, with nine OpenAI researchers, posts "Scaling Laws for Neural Language Models" to arXiv on January 23, 2020. The paper measures how a language model's error changes as its size, data, and compute vary.
Why it mattered
Error falls as a power law in each of the three, predictably, across several orders of magnitude. That makes it possible to work out in advance how well a bigger model will perform.
The full story
Choosing a model's size, how much data to feed it, and how long to train it had been largely a matter of trial and error. Kaplan's team trained a large number of Transformer language models of different sizes and measured their loss, the error they make when predicting text.
What came out was a simple rule: loss falls as a power law with model size, with the amount of data, and with training compute, and some of the trends hold across more than seven orders of magnitude. Other details, such as how wide or deep the network is, barely matter within a broad range. Simple equations also describe when overfitting sets in and how quickly a model trains.
That leads to practical advice. Large models make much better use of each piece of data, so for a fixed compute budget, the efficient choice is to train a very large model on a relatively modest amount of data and stop well before it converges.
Four months later, the GPT-3 paper leaned on these laws to justify its size: loss should keep improving smoothly as the model grew. In 2022, DeepMind's Chinchilla revised the balance between parameters and data that Kaplan had proposed.
Under the hood
The laws take the form L(N) ≈ (N_c / N)^α_N, where L is the loss and N is the number of parameters, not counting the word embeddings. Data D and compute C behave the same way, each with its own exponent: about 0.076 for parameters, 0.095 for data, and 0.050 for compute.
An exponent that small means diminishing returns. With α_N = 0.076, halving the loss would take roughly 9,000 times as many parameters. The rule holds as long as none of the three factors is holding back the others.
Common misconceptions
What people often say: Scaling laws say all you need is bigger models.
In fact: They describe the error in predicting text, not specific abilities, and only while parameters, data, and compute grow together. In 2022, Chinchilla showed it paid to use more data than Kaplan had proposed.
Leads to
Chinchilla (March 29, 2022)
Documented direct influence
Chinchilla contradicts and corrects the ratio Kaplan proposed
GPT-3 (May 28, 2020)
Documented direct influence
The GPT-3 paper justifies its size with scaling laws, which predict that validation loss improves as a smooth power law in model size (sections 1 and 2.1)
Anthropic is founded (January 2021)
Documented direct influence
The announcement of Anthropic's first funding round presents its team as the authors of earlier research on GPT-3, scaling laws, and learning from human preferences
Threads that pass through here
- Scale as a method (step 4 of 8)
- Who pays for compute (step 4 of 6)
Sources
Original source
- Scaling Laws for Neural Language Models (arXiv:2001.08361) (opens in a new tab). Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei, 2020
- p. 3 (opens in a new tab) Figure 1: loss falls as a power law with compute, data, and model size.
Further reading
How to cite
“Scaling laws.” The History of AI, revised November 13, 2026. https://thehistoryofai.com/en/milestone/scaling-laws
This entry was last reviewed on November 13, 2026.