ULMFiT
Era 4 · Deep learning and foundation models (2012–2021) › Transformers and language models
- Lineage
- Neurons and networks
- Kind
- Publication
Who
fast.ai, National University of Ireland, Galway
What happened
Jeremy Howard of fast.ai and Sebastian Ruder of the Insight Centre at the National University of Ireland, Galway, post a method to arXiv on January 18, 2018, for fine-tuning a pretrained language model to classify text. By the ACL 2018 version it is called ULMFiT.
Why it mattered
It brings to language the transfer learning that already worked in vision: pretrain once, then fine-tune for each task with only a few examples.
The full story
In computer vision, it was standard practice to start from a network already trained on ImageNet and fine-tune it for the task at hand. Language processing had nothing like it: as the paper's abstract puts it, the methods of the day still needed task-specific changes and training from scratch.
Howard and Ruder proposed a general, step-by-step method. First, a language model learns to predict the next word from a broad corpus. Next, that model is fine-tuned on text from the target task. Finally, a classifier goes on top and the fine-tuning is completed. To keep the model from forgetting what it had learned along the way, they introduced techniques such as unfreezing the layers gradually and giving each layer its own learning rate.
The first version, called FitLaM, improved the state of the art on five text classification tasks. The final one, by then named ULMFiT, did so on six and cut the error by 18 to 24 percent on most of the datasets. With just 100 labeled examples, it matched a model trained from scratch on a hundred times more data. The authors released their pretrained models and code.
A few months later, the GPT-1 paper cited Howard and Ruder among the work closest to its own: pretraining a language model and then fine-tuning it with supervision. ELMo, GPT-1, and BERT carried that recipe to other models and to many more tasks.
Leads to
ELMo, GPT-1, and BERT (February 15, 2018)
Documented direct influence
The GPT-1 paper cites Howard and Ruder among the work closest to its own: pretraining with a language-modeling objective and then fine-tuning with supervision (section 2)
Threads that pass through here
- Scale as a method (step 1 of 8)
Sources
Original source
- Universal Language Model Fine-tuning for Text Classification (arXiv:1801.06146) (opens in a new tab). Jeremy Howard and Sebastian Ruder, 2018
- Improving Language Understanding by Generative Pre-Training (opens in a new tab). Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever, 2018
- p. 4 (opens in a new tab) Figure 1: the Transformer that gets pretrained, and how each task's input is adapted for fine-tuning.
Scholarship and context
- Universal Language Model Fine-tuning for Text Classification. Proceedings of ACL 2018, vol. 1, pp. 328-339 (opens in a new tab). Jeremy Howard and Sebastian Ruder, 2018. open license: CC BY 4.0
How to cite
“ULMFiT.” The History of AI, revised November 13, 2026. https://thehistoryofai.com/en/milestone/ulmfit
This entry was last reviewed on November 13, 2026.