The History of AI

Scale as a method

From pretraining on unlabeled text to foundation models, by way of the laws that predict how much a model improves as it grows.

  1. ULMFiT (January 18, 2018)

    ULMFiT shows that a pretrained language model can be fine-tuned to classify text with few examples.

  2. ELMo, GPT-1, and BERT (February 15, 2018)

    ELMo, GPT-1, and BERT set the recipe of pretraining on lots of text and fine-tuning for each task.

  3. GPT-2 (February 14, 2019)

    GPT-2 follows GPT-1's design with more parameters and more text, and solves tasks with no fine-tuning at all.

  4. Scaling laws (January 23, 2020)

    Kaplan and colleagues measure that error falls predictably with size, data, and compute.

  5. GPT-3 (May 28, 2020)

    GPT-3 uses GPT-2's architecture at a hundred times the scale, justified by the scaling laws, and learns new tasks from a few examples.

  6. Scale reaches China and Korea (April 26, 2021)

    Huawei, the Beijing Academy of Artificial Intelligence, and Naver train GPT-3-sized models in Chinese and Korean.

  7. The term "foundation model" (August 16, 2021)

    Stanford calls models like BERT and GPT-3, which serve as the base for many tasks, "foundation models."

  8. Chinchilla (March 29, 2022)

    In 2022, Chinchilla corrects the balance between parameters and data that Kaplan had proposed.

Open in the interactive timeline