The History of AI

word2vec

Era 4 · Deep learning and foundation models (2012–2021) › Deep learning

Lineage
Neurons and networks
Kind
Publication

Who

Google

What happened

Tomáš Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean of Google post two architectures on arXiv for computing word vectors from very large corpora. Together with a second paper from the same year they make up word2vec, which Google releases in July 2013.

Why it mattered

It represents each word as a vector whose distances reflect relationships of meaning, with far more accuracy and far less computation than earlier models.

The full story

To a computer, a word was a symbol with no relation to any other. Neural language models, such as Bengio's in 2003 or Mikolov's own recurrent models in 2010, already learned vectors that brought similar words closer together, but they were expensive to train on large amounts of text.

Mikolov and colleagues stripped the problem down to the bare minimum. They proposed two architectures: CBOW, which guesses a word from the words around it, like a fill-in-the-blank exercise, and skip-gram, which does the reverse and predicts the context from the word. CBOW is faster, and skip-gram works better with rare words. According to the abstract, they achieved large improvements in accuracy at a much lower computational cost.

The resulting vectors captured relationships in surprising ways: by subtracting and adding vectors, they solved analogies such as "man is to woman as brother is to sister." A second paper, presented at NIPS 2013, improved the method and received the NeurIPS Test of Time Award in 2023.

Nobody fully explained why it worked so well, and later studies attributed part of its advantage to the choice of hyperparameters. Even so, pretrained word vectors were for years the standard way of feeding text to a neural network.

Builds on

Leads to

  • ELMo, GPT-1, and BERT (February 15, 2018)

    Documented direct influence

    ELMo starts from pretrained word vectors (citing Mikolov et al., 2013) and proposes replacing them with representations that depend on context (introduction)

Sources

Original source

How to cite

“word2vec.” The History of AI, revised November 13, 2026. https://thehistoryofai.com/en/milestone/word2vec

This entry was last reviewed on November 13, 2026.

Open in the interactive timeline