GPT-2
Era 4 · Deep learning and foundation models (2012–2021) › Transformers and language models
- Lineage
- Neurons and networks
- Kind
- Implementation
Who
OpenAI
What happened
OpenAI unveils GPT-2 on February 14, 2019: a 1.5-billion-parameter Transformer language model trained to predict the next word across 40 GB of internet text. The full model is not released until November 5.
Why it mattered
A large language model handles tasks nobody trained it for, setting the state of the art on 7 of 8 language modeling benchmarks without any fine-tuning. OpenAI releases it in stages, citing concerns about malicious use.
The full story
GPT-2 follows GPT-1's design closely, with a handful of changes, but it scales up both the model and the text. It has 1.5 billion parameters and is trained on WebText, some 8 million web pages collected by OpenAI, with a single objective: predict the next word.
What stood out was what the model could do without being taught. With no task-specific fine-tuning at all, it reached the state of the art on 7 of the 8 language modeling benchmarks it was tested on. The paper's title sums it up: language models are unsupervised multitask learners.
The launch was also a decision about what to release. OpenAI wrote that it was holding back the trained model out of concern about malicious applications, and chose a staged release instead: first a small version with 124 million parameters, then one with 355 million in May and one with 774 million on August 20, while the full model was shared only with research partners. The complete 1.5-billion-parameter model came out on November 5, 2019.
A year later, GPT-3 used the same model and architecture as GPT-2, with more than a hundred times as many parameters.
Common misconceptions
What people often say: OpenAI never released GPT-2 because it was too dangerous.
In fact: It released it in stages, smallest to largest, over the course of 2019. The full 1.5-billion-parameter model came out on November 5.
Builds on
ELMo, GPT-1, and BERT (February 15, 2018)
Documented direct influence
The GPT-2 model "largely follows the details of the OpenAI GPT model (Radford et al., 2018) with a few modifications" (section 2.3)
Leads to
GPT-3 (May 28, 2020)
Documented direct influence
"We use the same model and architecture as GPT-2" (section 2.1)
Threads that pass through here
- Scale as a method (step 3 of 8)
Sources
Original source
- Better language models and their implications (OpenAI, February 14, 2019) (opens in a new tab). OpenAI, 2019
- Language Models are Unsupervised Multitask Learners (opens in a new tab). Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever, 2019
- p. 3 (opens in a new tab) Section 2.1: WebText, the collection of web pages it is trained on.
- GPT-2: 1.5B release (OpenAI, November 5, 2019) (opens in a new tab). OpenAI, 2019
Scholarship and context
Further reading
How to cite
“GPT-2.” The History of AI, revised November 13, 2026. https://thehistoryofai.com/en/milestone/gpt-2
This entry was last reviewed on November 13, 2026.