The neural probabilistic language model
Era 3 · From the expert boom to the statistical turn (1980–2011) › Statistical learning and probabilistic models
- Lineage
- Data and learning
- Kind
- Publication
Who
Université de Montréal
What happened
Yoshua Bengio, Réjean Ducharme, Pascal Vincent, and Christian Jauvin of the University of Montreal publish "A Neural Probabilistic Language Model" in the Journal of Machine Learning Research. An early version was presented at the NIPS 2000 conference.
Why it mattered
The first neural language model to learn, at the same time, a vector for each word and the probability of the next word. It is the direct forerunner of word2vec and of neural language models.
The full story
A language model estimates which word is likely to come next. Those of the day were n-gram models: they counted how often each sequence of two, three, or four words appeared in a corpus. Their problem is the curse of dimensionality: most sentences encountered in practice never appeared in training, and the model does not know what to do with them.
Bengio and colleagues proposed representing each word as a point in a space of dozens of dimensions and having the network learn those positions while it learned to predict. If "dog" and "cat" appear in similar contexts, their vectors end up close together, and what is learned from a sentence about dogs carries over to sentences about cats that were never seen.
On two text corpora, the model clearly beat the best n-gram models and made use of longer contexts.
The idea of distributed word representations came back in a big way with word2vec in 2013 and underlies every language model today, all of which still learn vectors for each piece of text. The ACM cited this work in awarding Bengio the 2018 Turing Award, announced in 2019.
Leads to
word2vec (January 16, 2013)
Documented direct influence
An acknowledged forerunner
Sources
Original source
- A Neural Probabilistic Language Model. Journal of Machine Learning Research, 3, pp. 1137-1155 (opens in a new tab). Yoshua Bengio, Réjean Ducharme, Pascal Vincent, and Christian Jauvin, 2003
How to cite
“The neural probabilistic language model.” The History of AI, revised November 13, 2026. https://thehistoryofai.com/en/milestone/bengio-neural-language-model
This entry was last reviewed on November 13, 2026.