Inverse document frequency
Era 2 · AI is born (1943–1979) › Symbolic AI
- Lineage
- Data and learning
- Kind
- Publication
Who
University of Cambridge
What happened
Karen Spärck Jones of the University of Cambridge publishes "A Statistical Interpretation of Term Specificity and Its Application in Retrieval" in the Journal of Documentation.
Why it mattered
It proposes giving each word a weight that grows the rarer the word is in the document collection, the inverse document frequency (IDF). Combined with how often the word appears in each text, it was the basis of search engines for decades.
The full story
A document retrieval system compares the words of a query with those of each text. If every word counts the same, very common words, which appear in almost every document, dominate the result even though they do not tell one text from another.
Spärck Jones proposed a statistical solution. A word that appears in few documents is more specific and says more about them than one that appears in all of them. Its weight should grow as the proportion of documents containing it shrinks. Combined with the word's frequency within each document, that weight gives the score known as TF-IDF, which is used to rank documents by their relevance to a query.
Spärck Jones had trained at the Cambridge Language Research Unit, the computational linguistics group led by Margaret Masterman, and spent a whole career in language processing and information retrieval. Spärck Jones also championed women in computing with an often-quoted line: "Computing is too important to be left to men."
TF-IDF weighting was for decades the basis of search engines and of many text classification systems, and the idea that a word's rarity measures how informative it is lives on in language processing.
Sources
Original source
Further reading
How to cite
“Inverse document frequency.” The History of AI, revised November 13, 2026. https://thehistoryofai.com/en/milestone/sparck-jones-inverse-document-frequency
This entry was last reviewed on November 13, 2026.