TF-IDF and BM25 score relevance as exact probability differences
TF-IDF and BM25 Are Exact KL Divergences
Information RetrievalComputation and Language
Summary
Scoring methods like TF-IDF and BM25 help computers find documents matching a search query. Until now, there was no clear probability-based reason why these scores work so well. The authors show that both methods can be exactly seen as measuring differences between probability models, known as Kullback-Leibler divergences. This gives a unified mathematical explanation for these popular techniques and helps compare them with other search algorithms.
What this means in practice
- •For search engine engineers: Design scoring algorithms with a clear probabilistic basis to improve or analyze search relevance scoring.
- •For document classification teams: Compare TF-IDF and BM25 to other probabilistic models to choose the best statistical features for text classification.
A theory result. No direct application yet.
Authors
Ivan Silajev
Abstract
TF-IDF and BM25 are two of the most widely used methods for scoring query-document relevance, yet neither has a standard probabilistic derivation that justifies it as a statistical method within a unified framework. We address this gap by showing that both scoring methods admit an exact interpretation as Kullback-Leibler divergences between two probability models. We treat the BM25 variant that includes the plus 1 correction in the IDF term, which is the one used in practice, and also discuss the original BM25 formulation without that correction. The resulting framework provides a common theoretical basis for TF-IDF and BM25, clarifies what they measure, and allows them to be compared theoretically with other information retrieval methods rather than only experimentally.