ELMo, GPT-1, and BERT
Era 4 · Deep learning and foundation models (2012–2021) › Transformers and language models
- Lineage
- Neurons and networks
- Kind
- Publication
Who
Allen Institute for AI, University of Washington, OpenAI, Google
What happened
Three papers from 2018 show how to pretrain a model on unlabeled text and then put it to work on specific tasks: ELMo, by Matthew E. Peters and colleagues (on arXiv February 15), OpenAI's GPT-1 (June 11), and Google's BERT (on arXiv October 11).
Why it mattered
They set the recipe that later language models follow: long, general pretraining, then a short round of fine-tuning for each task. BERT, the most complete of the three, sets a new state of the art on eleven tasks.
The full story
Until 2018, each language task (answering questions, classifying opinions, spotting names) usually got its own model, trained on data labeled for that task alone, which is expensive and scarce. In January, ULMFiT had shown that a pretrained language model could be fine-tuned to classify text. The three papers that followed tapped the same resource: ordinary, unlabeled text, of which there is an enormous supply.
ELMo, from the Allen Institute for AI and the University of Washington, gives every word a representation that depends on the sentence around it, drawn from a bidirectional language model trained on a large corpus. Plugged into existing models, it improved the state of the art on six tasks, with relative error reductions of up to 20 percent.
OpenAI's GPT-1 brings together two ideas, the Transformer and unsupervised pretraining. First a Transformer learns to predict text across more than 7,000 unpublished books; then it is fine-tuned on small labeled datasets. It beat the state of the art on 9 of the 12 tasks studied.
Google's BERT changes how the pretraining works. GPT-1 only looks at the words that come before; BERT looks both ways at once, because it learns by guessing words masked out in the middle of a sentence and by deciding whether one sentence follows another. With a single extra layer for each task, it set a new state of the art on eleven of them and pushed the GLUE score to 80.5 percent, a gain of 7.7 points. It won the best long paper award at NAACL 2019.
That pattern, pretrain first and fine-tune after, is what GPT-2 and GPT-3 would take to a far larger scale.
Builds on
The Transformer (June 12, 2017)
Documented direct influence
GPT-1 combines Transformers and unsupervised pretraining, and BERT's architecture is "a multi-layer bidirectional Transformer encoder" based on the original implementation by Vaswani et al.
word2vec (January 16, 2013)
Documented direct influence
ELMo starts from pretrained word vectors (citing Mikolov et al., 2013) and proposes replacing them with representations that depend on context (introduction)
ULMFiT (January 18, 2018)
Documented direct influence
The GPT-1 paper cites Howard and Ruder among the work closest to its own: pretraining with a language-modeling objective and then fine-tuning with supervision (section 2)
Leads to
GPT-2 (February 14, 2019)
Documented direct influence
The GPT-2 model "largely follows the details of the OpenAI GPT model (Radford et al., 2018) with a few modifications" (section 2.3)
The term "foundation model" (August 16, 2021)
Documented direct influence
The report coins the term for models like BERT and GPT-3, which are trained on broad data and adapted to many tasks, and names them as examples from its first sentence
Scale reaches China and Korea (April 26, 2021)
Documented direct influence
ERNIE draws on BERT's masking strategy, and PanGu-α and HyperCLOVA are presented as building on GPT-3 (the abstracts of all three papers)
Threads that pass through here
- Scale as a method (step 2 of 8)
Sources
Original source
- Deep Contextualized Word Representations. NAACL-HLT 2018, pp. 2227-2237 (opens in a new tab). Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer, 2018. open license: CC BY 4.0
- Improving language understanding with unsupervised learning (OpenAI, June 11, 2018) (opens in a new tab). OpenAI, 2018
- Improving Language Understanding by Generative Pre-Training (opens in a new tab). Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever, 2018
- p. 4 (opens in a new tab) Figure 1: the Transformer that gets pretrained, and how each task's input is adapted for fine-tuning.
- BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding (arXiv:1810.04805) (opens in a new tab). Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova, 2018
- BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. NAACL-HLT 2019, pp. 4171-4186 (opens in a new tab). Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova, 2019. open license: CC BY 4.0
How to cite
“ELMo, GPT-1, and BERT.” The History of AI, revised November 13, 2026. https://thehistoryofai.com/en/milestone/elmo-gpt-1-and-bert
This entry was last reviewed on November 13, 2026.