The History of AI

Chinchilla

Era 5 · Generative AI, reasoning, and agents (2022–2026) › Infrastructure and compute

Lineage
Machines and computation
Kind
Publication

Who

DeepMind

What happened

Jordan Hoffmann and colleagues at DeepMind show that large language models were undertrained: for a fixed compute budget, model size and the amount of data should grow at the same rate. Their model Chinchilla, with 70 billion parameters, outperforms others four times its size.

Why it mattered

It corrects the 2020 scaling laws: parameters and data should grow in similar proportion. It changes how compute is divided in the models that follow.

The full story

Kaplan's scaling laws, from 2020, had led the labs to build ever larger models without increasing training data nearly as much. GPT-3 had 175 billion parameters, DeepMind's Gopher 280 billion, and Megatron-Turing NLG 530 billion.

Hoffmann and colleagues trained more than 400 language models, from 70 million to more than 16 billion parameters, on 5 billion to 500 billion tokens, to measure how best to divide a compute budget. Their conclusion was that model size and data should grow together: every time the size doubles, the data should double too. The large models of the day were therefore undertrained.

To test it they trained Chinchilla, with 70 billion parameters and 1.4 trillion tokens, four times as much data as Gopher with the same compute. Chinchilla outperformed Gopher, GPT-3, Jurassic-1, and Megatron-Turing NLG on many tasks, and on MMLU, a test of knowledge across many subjects, it reached an average of 67.5 percent, more than seven points above Gopher.

A smaller model with the same performance is also cheaper to use, because each response takes less computation. The lesson for the labs was that data mattered as much as size.

Two pairs of bars. Gopher has a long parameter bar, 280 billion, and a short token bar, 300 billion. Chinchilla has a short parameter bar, 70 billion, and a long token bar, 1.4 trillion.

With the same compute, Chinchilla uses four times fewer parameters than Gopher and four times more data.

Under the hood

The compute needed to train a model grows with the product of its parameters N and its training tokens D: roughly C ≈ 6·N·D operations. For a fixed C, the question is how to divide it between N and D.

The authors measured this in three different ways, and all three showed that N and D should grow in the same proportion with compute, roughly as C^0.5 each. Kaplan had estimated that size should grow much faster than data.

In practice, the rule comes to about 20 tokens per parameter: Chinchilla, with 70 billion parameters, was trained on 1.4 trillion tokens. Gopher, with 280 billion, had seen only 300 billion, about one token per parameter.

Side by side

Gopher and Chinchilla, with the same compute
MeasureGopherChinchilla
Parameters280 billion70 billion
Training tokens300 billion1.4 trillion
MMLU (average)60.0 percent67.5 percent

Builds on

  • Scaling laws (January 23, 2020)

    Documented direct influence

    Chinchilla contradicts and corrects the ratio Kaplan proposed

Leads to

  • LLaMA (February 24, 2023)

    Documented direct influence

    LLaMA starts from the scaling laws of Hoffmann et al. (2022) and chooses to train smaller models on more tokens than those laws recommend, with the cost of running them in mind (section 1)

Threads that pass through here

Sources

How to cite

“Chinchilla.” The History of AI, revised November 13, 2026. https://thehistoryofai.com/en/milestone/chinchilla

This entry was last reviewed on November 13, 2026.

Open in the interactive timeline