Scale as a method
From pretraining on unlabeled text to foundation models, by way of the laws that predict how much a model improves as it grows.
ULMFiT (January 18, 2018)
ULMFiT shows that a pretrained language model can be fine-tuned to classify text with few examples.
ELMo, GPT-1, and BERT (February 15, 2018)
ELMo, GPT-1, and BERT set the recipe of pretraining on lots of text and fine-tuning for each task.
GPT-2 (February 14, 2019)
GPT-2 follows GPT-1's design with more parameters and more text, and solves tasks with no fine-tuning at all.
Scaling laws (January 23, 2020)
Kaplan and colleagues measure that error falls predictably with size, data, and compute.
GPT-3 (May 28, 2020)
GPT-3 uses GPT-2's architecture at a hundred times the scale, justified by the scaling laws, and learns new tasks from a few examples.
Scale reaches China and Korea (April 26, 2021)
Huawei, the Beijing Academy of Artificial Intelligence, and Naver train GPT-3-sized models in Chinese and Korean.
The term "foundation model" (August 16, 2021)
Stanford calls models like BERT and GPT-3, which serve as the base for many tasks, "foundation models."
Chinchilla (March 29, 2022)
In 2022, Chinchilla corrects the balance between parameters and data that Kaplan had proposed.