Chinchilla
Era 5 · Generative AI, reasoning, and agents (2022–2026) › Infrastructure and compute
- Lineage
- Machines and computation
- Kind
- Publication
Who
DeepMind
What happened
Jordan Hoffmann and colleagues at DeepMind show that large language models were undertrained: for a fixed compute budget, model size and the amount of data should grow at the same rate. Their model Chinchilla, with 70 billion parameters, outperforms others four times its size.
Why it mattered
It corrects the 2020 scaling laws: parameters and data should grow in similar proportion. It changes how compute is divided in the models that follow.
The full story
Kaplan's scaling laws, from 2020, had led the labs to build ever larger models without increasing training data nearly as much. GPT-3 had 175 billion parameters, DeepMind's Gopher 280 billion, and Megatron-Turing NLG 530 billion.
Hoffmann and colleagues trained more than 400 language models, from 70 million to more than 16 billion parameters, on 5 billion to 500 billion tokens, to measure how best to divide a compute budget. Their conclusion was that model size and data should grow together: every time the size doubles, the data should double too. The large models of the day were therefore undertrained.
To test it they trained Chinchilla, with 70 billion parameters and 1.4 trillion tokens, four times as much data as Gopher with the same compute. Chinchilla outperformed Gopher, GPT-3, Jurassic-1, and Megatron-Turing NLG on many tasks, and on MMLU, a test of knowledge across many subjects, it reached an average of 67.5 percent, more than seven points above Gopher.
A smaller model with the same performance is also cheaper to use, because each response takes less computation. The lesson for the labs was that data mattered as much as size.
Two pairs of bars. Gopher has a long parameter bar, 280 billion, and a short token bar, 300 billion. Chinchilla has a short parameter bar, 70 billion, and a long token bar, 1.4 trillion.
Under the hood
The compute needed to train a model grows with the product of its parameters N and its training tokens D: roughly C ≈ 6·N·D operations. For a fixed C, the question is how to divide it between N and D.
The authors measured this in three different ways, and all three showed that N and D should grow in the same proportion with compute, roughly as C^0.5 each. Kaplan had estimated that size should grow much faster than data.
In practice, the rule comes to about 20 tokens per parameter: Chinchilla, with 70 billion parameters, was trained on 1.4 trillion tokens. Gopher, with 280 billion, had seen only 300 billion, about one token per parameter.
Side by side
| Measure | Gopher | Chinchilla |
|---|---|---|
| Parameters | 280 billion | 70 billion |
| Training tokens | 300 billion | 1.4 trillion |
| MMLU (average) | 60.0 percent | 67.5 percent |
Builds on
Scaling laws (January 23, 2020)
Documented direct influence
Chinchilla contradicts and corrects the ratio Kaplan proposed
Leads to
LLaMA (February 24, 2023)
Documented direct influence
LLaMA starts from the scaling laws of Hoffmann et al. (2022) and chooses to train smaller models on more tokens than those laws recommend, with the cost of running them in mind (section 1)
Threads that pass through here
- Scale as a method (step 8 of 8)
Sources
Original source
- Training Compute-Optimal Large Language Models (arXiv:2203.15556) (opens in a new tab). Jordan Hoffmann et al., 2022
How to cite
“Chinchilla.” The History of AI, revised November 13, 2026. https://thehistoryofai.com/en/milestone/chinchilla
This entry was last reviewed on November 13, 2026.