The History of AI

GPT-3

Era 4 · Deep learning and foundation models (2012–2021) › Transformers and language models

Lineage
Neurons and networks
Kind
Implementation

Who

OpenAI

What happened

OpenAI posts "Language Models are Few-Shot Learners" to arXiv on May 28, 2020, with 31 authors led by Tom B. Brown. It introduces GPT-3, a language model with 175 billion parameters, ten times more than any earlier non-sparse language model.

Why it mattered

In-context learning: GPT-3 takes on new tasks from an instruction and a few examples written into the prompt, without any change to its weights.

The full story

GPT-3 uses the same model and architecture as GPT-2, with one exception in how attention is arranged. What changes is the scale: 175 billion parameters, more than a hundred times GPT-2's, trained on 300 billion tokens, the small chunks of text a model reads. Sixty percent of the data comes from a filtered version of Common Crawl, an archive of the web; the rest comes from books, English Wikipedia, and an expanded WebText. The authors rely on the scaling laws published months earlier, which predicted that loss would keep improving smoothly as the model grew.

But the paper's real question is a different one. GPT-1 and BERT had to be fine-tuned on labeled examples for every task. GPT-3 is tested without touching its weights: the task is described in the text itself, with no examples, one example, or a few, and the model simply carries on. The paper calls this few-shot learning, and its Figure 2.1 sets it against traditional fine-tuning.

The paper also flags a risk: GPT-3 wrote news articles that human evaluators had trouble telling apart from ones written by people.

In 2021, Codex, a GPT model fine-tuned on code, became the engine of GitHub Copilot, and in 2022 InstructGPT started from GPT-3 and trained it on human preferences.

Side by side

From GPT-1 to GPT-3
FeatureGPT-1 (2018)GPT-2 (2019)GPT-3 (2020)
ParametersAbout 120 million1.5 billion175 billion
Layers124896
Training dataBooksCorpus: over 7,000 unpublished booksWebText: about 8 million web pages300 billion tokens from filtered Common Crawl, WebText2, books, and Wikipedia
How it takes on a taskFine-tuned on labeled examplesTested with no fine-tuningGiven the task and a few examples in the prompt

Builds on

  • GPT-2 (February 14, 2019)

    Documented direct influence

    "We use the same model and architecture as GPT-2" (section 2.1)

  • Scaling laws (January 23, 2020)

    Documented direct influence

    The GPT-3 paper justifies its size with scaling laws, which predict that validation loss improves as a smooth power law in model size (sections 1 and 2.1)

  • OpenAI adopts a capped-profit model (March 11, 2019)

    Documented direct influence

    GPT-3 was trained on V100 GPUs in "a high-bandwidth cluster provided by Microsoft" (section 2.3), the partner that came in with the 2019 investment

Leads to

  • GitHub Copilot (June 29, 2021)

    Documented direct influence

    Copilot runs on Codex, a GPT language model fine-tuned on code from GitHub. Its authors start from the GPT-3 family because fine-tuning converges faster, and a production version of Codex powers Copilot

  • The term "foundation model" (August 16, 2021)

    Documented direct influence

    The report coins the term for models like BERT and GPT-3, which are trained on broad data and adapted to many tasks, and names them as examples from its first sentence

  • InstructGPT (March 4, 2022)

    Documented direct influence

    InstructGPT is GPT-3 fine-tuned first on demonstrations written by people and then with reinforcement learning from their preferences (abstract)

  • Chain of thought (January 28, 2022)

    Documented direct influence

    GPT-3 is one of the three model families the paper tests chain of thought on, and the baseline it beats on GSM8K is GPT-3 fine-tuned with a verifier (abstract and section 3)

  • Anthropic is founded (January 2021)

    Documented direct influence

    The announcement of Anthropic's first funding round presents its team as the authors of earlier research on GPT-3, scaling laws, and learning from human preferences

  • Scale reaches China and Korea (April 26, 2021)

    Documented direct influence

    ERNIE draws on BERT's masking strategy, and PanGu-α and HyperCLOVA are presented as building on GPT-3 (the abstracts of all three papers)

Threads that pass through here

Sources

Original source

How to cite

“GPT-3.” The History of AI, revised November 13, 2026. https://thehistoryofai.com/en/milestone/gpt-3

This entry was last reviewed on November 13, 2026.

Open in the interactive timeline