word2vec
Era 4 · Deep learning and foundation models (2012–2021) › Deep learning
- Lineage
- Neurons and networks
- Kind
- Publication
Who
What happened
Tomáš Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean of Google post two architectures on arXiv for computing word vectors from very large corpora. Together with a second paper from the same year they make up word2vec, which Google releases in July 2013.
Why it mattered
It represents each word as a vector whose distances reflect relationships of meaning, with far more accuracy and far less computation than earlier models.
The full story
To a computer, a word was a symbol with no relation to any other. Neural language models, such as Bengio's in 2003 or Mikolov's own recurrent models in 2010, already learned vectors that brought similar words closer together, but they were expensive to train on large amounts of text.
Mikolov and colleagues stripped the problem down to the bare minimum. They proposed two architectures: CBOW, which guesses a word from the words around it, like a fill-in-the-blank exercise, and skip-gram, which does the reverse and predicts the context from the word. CBOW is faster, and skip-gram works better with rare words. According to the abstract, they achieved large improvements in accuracy at a much lower computational cost.
The resulting vectors captured relationships in surprising ways: by subtracting and adding vectors, they solved analogies such as "man is to woman as brother is to sister." A second paper, presented at NIPS 2013, improved the method and received the NeurIPS Test of Time Award in 2023.
Nobody fully explained why it worked so well, and later studies attributed part of its advantage to the choice of hyperparameters. Even so, pretrained word vectors were for years the standard way of feeding text to a neural network.
Builds on
The neural probabilistic language model (February 2003)
Documented direct influence
An acknowledged forerunner
Leads to
ELMo, GPT-1, and BERT (February 15, 2018)
Documented direct influence
ELMo starts from pretrained word vectors (citing Mikolov et al., 2013) and proposes replacing them with representations that depend on context (introduction)
Sources
Original source
- Efficient Estimation of Word Representations in Vector Space (arXiv:1301.3781) (opens in a new tab). Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean, 2013
Further reading
How to cite
“word2vec.” The History of AI, revised November 13, 2026. https://thehistoryofai.com/en/milestone/word2vec
This entry was last reviewed on November 13, 2026.