GPT-3
Era 4 · Deep learning and foundation models (2012–2021) › Transformers and language models
- Lineage
- Neurons and networks
- Kind
- Implementation
Who
OpenAI
What happened
OpenAI posts "Language Models are Few-Shot Learners" to arXiv on May 28, 2020, with 31 authors led by Tom B. Brown. It introduces GPT-3, a language model with 175 billion parameters, ten times more than any earlier non-sparse language model.
Why it mattered
In-context learning: GPT-3 takes on new tasks from an instruction and a few examples written into the prompt, without any change to its weights.
The full story
GPT-3 uses the same model and architecture as GPT-2, with one exception in how attention is arranged. What changes is the scale: 175 billion parameters, more than a hundred times GPT-2's, trained on 300 billion tokens, the small chunks of text a model reads. Sixty percent of the data comes from a filtered version of Common Crawl, an archive of the web; the rest comes from books, English Wikipedia, and an expanded WebText. The authors rely on the scaling laws published months earlier, which predicted that loss would keep improving smoothly as the model grew.
But the paper's real question is a different one. GPT-1 and BERT had to be fine-tuned on labeled examples for every task. GPT-3 is tested without touching its weights: the task is described in the text itself, with no examples, one example, or a few, and the model simply carries on. The paper calls this few-shot learning, and its Figure 2.1 sets it against traditional fine-tuning.
The paper also flags a risk: GPT-3 wrote news articles that human evaluators had trouble telling apart from ones written by people.
In 2021, Codex, a GPT model fine-tuned on code, became the engine of GitHub Copilot, and in 2022 InstructGPT started from GPT-3 and trained it on human preferences.
Side by side
| Feature | GPT-1 (2018) | GPT-2 (2019) | GPT-3 (2020) |
|---|---|---|---|
| Parameters | About 120 million | 1.5 billion | 175 billion |
| Layers | 12 | 48 | 96 |
| Training data | BooksCorpus: over 7,000 unpublished books | WebText: about 8 million web pages | 300 billion tokens from filtered Common Crawl, WebText2, books, and Wikipedia |
| How it takes on a task | Fine-tuned on labeled examples | Tested with no fine-tuning | Given the task and a few examples in the prompt |
Builds on
GPT-2 (February 14, 2019)
Documented direct influence
"We use the same model and architecture as GPT-2" (section 2.1)
Scaling laws (January 23, 2020)
Documented direct influence
The GPT-3 paper justifies its size with scaling laws, which predict that validation loss improves as a smooth power law in model size (sections 1 and 2.1)
OpenAI adopts a capped-profit model (March 11, 2019)
Documented direct influence
GPT-3 was trained on V100 GPUs in "a high-bandwidth cluster provided by Microsoft" (section 2.3), the partner that came in with the 2019 investment
Leads to
GitHub Copilot (June 29, 2021)
Documented direct influence
Copilot runs on Codex, a GPT language model fine-tuned on code from GitHub. Its authors start from the GPT-3 family because fine-tuning converges faster, and a production version of Codex powers Copilot
The term "foundation model" (August 16, 2021)
Documented direct influence
The report coins the term for models like BERT and GPT-3, which are trained on broad data and adapted to many tasks, and names them as examples from its first sentence
InstructGPT (March 4, 2022)
Documented direct influence
InstructGPT is GPT-3 fine-tuned first on demonstrations written by people and then with reinforcement learning from their preferences (abstract)
Chain of thought (January 28, 2022)
Documented direct influence
GPT-3 is one of the three model families the paper tests chain of thought on, and the baseline it beats on GSM8K is GPT-3 fine-tuned with a verifier (abstract and section 3)
Anthropic is founded (January 2021)
Documented direct influence
The announcement of Anthropic's first funding round presents its team as the authors of earlier research on GPT-3, scaling laws, and learning from human preferences
Scale reaches China and Korea (April 26, 2021)
Documented direct influence
ERNIE draws on BERT's masking strategy, and PanGu-α and HyperCLOVA are presented as building on GPT-3 (the abstracts of all three papers)
Threads that pass through here
- From prompted reasoning to trained reasoning (step 1 of 9)
- Scale as a method (step 5 of 8)
- Who pays for compute (step 5 of 6)
Sources
Original source
- Language Models are Few-Shot Learners (arXiv:2005.14165) (opens in a new tab). Tom B. Brown, and 30 others, 2020
- p. 7 (opens in a new tab) Figure 2.1: zero-shot, one-shot, and few-shot learning in the prompt, compared with traditional fine-tuning.
Scholarship and context
Further reading
How to cite
“GPT-3.” The History of AI, revised November 13, 2026. https://thehistoryofai.com/en/milestone/gpt-3
This entry was last reviewed on November 13, 2026.