The History of AI

ULMFiT

Era 4 · Deep learning and foundation models (2012–2021) › Transformers and language models

Lineage
Neurons and networks
Kind
Publication

Who

fast.ai, National University of Ireland, Galway

What happened

Jeremy Howard of fast.ai and Sebastian Ruder of the Insight Centre at the National University of Ireland, Galway, post a method to arXiv on January 18, 2018, for fine-tuning a pretrained language model to classify text. By the ACL 2018 version it is called ULMFiT.

Why it mattered

It brings to language the transfer learning that already worked in vision: pretrain once, then fine-tune for each task with only a few examples.

The full story

In computer vision, it was standard practice to start from a network already trained on ImageNet and fine-tune it for the task at hand. Language processing had nothing like it: as the paper's abstract puts it, the methods of the day still needed task-specific changes and training from scratch.

Howard and Ruder proposed a general, step-by-step method. First, a language model learns to predict the next word from a broad corpus. Next, that model is fine-tuned on text from the target task. Finally, a classifier goes on top and the fine-tuning is completed. To keep the model from forgetting what it had learned along the way, they introduced techniques such as unfreezing the layers gradually and giving each layer its own learning rate.

The first version, called FitLaM, improved the state of the art on five text classification tasks. The final one, by then named ULMFiT, did so on six and cut the error by 18 to 24 percent on most of the datasets. With just 100 labeled examples, it matched a model trained from scratch on a hundred times more data. The authors released their pretrained models and code.

A few months later, the GPT-1 paper cited Howard and Ruder among the work closest to its own: pretraining a language model and then fine-tuning it with supervision. ELMo, GPT-1, and BERT carried that recipe to other models and to many more tasks.

Leads to

  • ELMo, GPT-1, and BERT (February 15, 2018)

    Documented direct influence

    The GPT-1 paper cites Howard and Ruder among the work closest to its own: pretraining with a language-modeling objective and then fine-tuning with supervision (section 2)

Threads that pass through here

Sources

Original source

Scholarship and context

How to cite

“ULMFiT.” The History of AI, revised November 13, 2026. https://thehistoryofai.com/en/milestone/ulmfit

This entry was last reviewed on November 13, 2026.

Open in the interactive timeline