The History of AI

The neural probabilistic language model

Era 3 · From the expert boom to the statistical turn (1980–2011) › Statistical learning and probabilistic models

Lineage
Data and learning
Kind
Publication

Who

Université de Montréal

What happened

Yoshua Bengio, Réjean Ducharme, Pascal Vincent, and Christian Jauvin of the University of Montreal publish "A Neural Probabilistic Language Model" in the Journal of Machine Learning Research. An early version was presented at the NIPS 2000 conference.

Why it mattered

The first neural language model to learn, at the same time, a vector for each word and the probability of the next word. It is the direct forerunner of word2vec and of neural language models.

The full story

A language model estimates which word is likely to come next. Those of the day were n-gram models: they counted how often each sequence of two, three, or four words appeared in a corpus. Their problem is the curse of dimensionality: most sentences encountered in practice never appeared in training, and the model does not know what to do with them.

Bengio and colleagues proposed representing each word as a point in a space of dozens of dimensions and having the network learn those positions while it learned to predict. If "dog" and "cat" appear in similar contexts, their vectors end up close together, and what is learned from a sentence about dogs carries over to sentences about cats that were never seen.

On two text corpora, the model clearly beat the best n-gram models and made use of longer contexts.

The idea of distributed word representations came back in a big way with word2vec in 2013 and underlies every language model today, all of which still learn vectors for each piece of text. The ACM cited this work in awarding Bengio the 2018 Turing Award, announced in 2019.

Leads to

  • word2vec (January 16, 2013)

    Documented direct influence

    An acknowledged forerunner

Sources

Original source

How to cite

“The neural probabilistic language model.” The History of AI, revised November 13, 2026. https://thehistoryofai.com/en/milestone/bengio-neural-language-model

This entry was last reviewed on November 13, 2026.

Open in the interactive timeline