BM25 vs TF-IDF: What Changed and Why It Matters

BM25 is a ranking function that calculates document relevance by scoring how frequently query terms appear while accounting for term saturation and document length. It replaced classic TF-IDF across modern search engines because TF-IDF scales linearly, rewarding keyword repetition and favoring excessively long documents. BM25 sets an upper bound on term frequency and penalizes documents that exceed the average corpus length.

Start with what TF-IDF does

Understanding BM25 requires examining the foundational information retrieval model that came before it. Classic TF-IDF measures the relevance of a document to a search query by multiplying two separate statistical values: term frequency and inverse document frequency. Term frequency counts how many times a given query word appears inside a target document. Inverse document frequency measures how rare that word is across the broader document collection.

The conceptual intuition behind TF-IDF is straightforward. If a user searches for the query “crawler architecture”, a document that mentions “crawler” multiple times is more likely to be about crawlers than a document that mentions the word only once. At the same time, common stop words like “the”, “and”, or “is” appear millions of times across every page on the web. Inverse document frequency dampens the weight of these ubiquitous words while elevating rare technical terms.

In standard TF-IDF formulations, term frequency is calculated as raw frequency or through a logarithmic scaling function. Inverse document frequency is computed by dividing the total number of documents in the collection by the number of documents that contain the target term, and then taking the logarithm of that ratio. You can explore how search engines store these word occurrences across document collections in our guide to how inverted indexes work.

While TF-IDF was a major advance for early digital libraries, it possesses severe structural flaws when applied to open-ended web collections. It treats text as a simple bag of words without considering text density, document verbosity, or unnatural term repetition. These flaws made early search engines vulnerable to manipulation and prompted researchers to develop superior probabilistic ranking algorithms.

The two things TF-IDF gets wrong

Classic TF-IDF breaks down in real-world search environments because of two mathematical assumptions. First, it assumes that relevance scales continuously with term frequency. Second, it assumes that document length does not distort the probability that a document is relevant. Examining both assumptions with concrete numbers illustrates why standard TF-IDF fails modern searchers.

The first failure is linear term frequency scaling. Suppose a user searches for the word “database”. Document A mentions “database” five times across a focused 300-word explanation.

Document B mentions “database” one hundred times across a rambling text that simply repeats the word in long keyword lists. Under classic linear TF-IDF, Document B receives twenty times the term frequency score of Document A. Repeating a word one hundred times does not make a document twenty times more helpful.

The second failure is bias toward long documents. A 10,000-word academic textbook naturally uses a broader vocabulary than a 500-word quick reference guide. Because long documents contain thousands of words, they accumulate higher raw term frequencies purely through natural verbosity. If a textbook mentions the query term twelve times across twenty chapters, classic TF-IDF ranks it ahead of a concise article where the term appears four times in a two-paragraph answer.

To observe this distortion in practice, consider a toy corpus of three documents. Document 1 is a 200-word tutorial mentioning “indexing” three times. Document 2 is a 2,000-word glossary mentioning “indexing” six times.

Document 3 is a 400-word spam page repeating “indexing” thirty times. Classic TF-IDF ranks Document 3 first, Document 2 second, and Document 1 last. The best answer ranks worst because TF-IDF cannot differentiate between natural topical relevance and sheer keyword accumulation.

Fix 1: term frequency saturation

The first fundamental innovation of BM25 is term frequency saturation. Rather than allowing term scores to increase indefinitely as a word is repeated, BM25 introduces an asymptotic curve. The first few occurrences of a query term provide a dramatic increase in the relevance score. As the term appears more frequently, each additional mention yields smaller and smaller score increments until the score reaches a rigid mathematical ceiling.

This behavior mirrors human reading comprehension. If an article about web crawling mentions the word “crawler” once in the opening paragraph, the reader learns that the page discusses crawlers. If the word appears three more times across distinct subheadings, the reader gains strong confidence that the entire document focuses on crawlers.

However, if the word appears another forty times, those additional mentions provide zero extra confirmation. The document cannot become more about crawlers than it already is.

The table below illustrates how raw term frequency compares to BM25 saturated term scoring as the term count increases from one to one hundred occurrences. Evaluating these figures demonstrates how rapidly the marginal value of keyword repetition decays.

The mathematical ceiling for saturated term frequency in this formulation is 1.0 plus the parameter ceiling. Going from one mention to two mentions increases the score by 40 percent. In contrast, going from twenty mentions to one hundred mentions increases the score by barely 5 percent. By implementing an asymptote, BM25 completely eliminates the incentive for web authors to engage in repetitive keyword stuffing.

Fix 2: document length normalization

The second major breakthrough of BM25 is document length normalization. Documents in a web collection vary wildly in length, ranging from fifty-word forum snippets to multi-volume reference books. BM25 evaluates the length of a specific document against the average document length across the entire collection.

Length normalization accounts for two competing reasons why a document might be long. First, a document might be long because it is verbose, using flowery language and redundant paragraphs to cover a single topic. Second, a document might be long because it covers a broad scope of multiple independent topics. If a document is simply verbose, the occurrence of a query term carries less information density than the same term appearing in a concise summary.

BM25 penalizes documents that are longer than the average document length of the collection, and rewards documents that are shorter than average. However, it does not do so linearly. Instead, it adjusts the rate at which term frequency saturates based on document length.

A short document saturates its term score very quickly with just a few mentions. A long document requires more mentions of the term to achieve the same saturation level because its word density is diluted.

The table below demonstrates how document length affects the normalized score for three documents that each contain the query term exactly three times. In this corpus, the average document length is 1,000 words.

| Document | Total Words (|D|) | Relative Length (|D| / avgdl) | Occurrences of Term | Effective BM25 TF (b = 0.75) | | :— | :— | :— | :— | :— | | Short Guide | 250 | 0.25 | 3 | 0.81 | | Average Article | 1,000 | 1.00 | 3 | 0.71 | | Long Manual | 4,000 | 4.00 | 3 | 0.44 |

Notice the dramatic difference in scoring. Even though all three documents contain the query term three times, the short guide receives a score of 0.81, while the long manual receives only 0.44.

In the 250-word guide, the query term represents more than one percent of the entire text, indicating intense topical focus. In the 4,000-word manual, the three mentions represent an incidental footnote. Document length normalization ensures that concise, highly relevant answers win over bloated reference pages.

The formula, one piece at a time

The complete BM25 formula looks intimidating when viewed as a single algebraic expression. However, the equation consists of three logical components multiplied together: the inverse document frequency factor, the saturated term frequency factor, and the document length normalization scaling factor. Breaking the formula down into these three pieces makes the underlying mechanism transparent.

The first component is inverse document frequency (IDF). BM25 uses an updated probabilistic IDF formula derived by Stephen Robertson and Karen Sparck Jones. Given a collection with N total documents where n(qᵢ) documents contain the query term qᵢ, the Robertson-Sparck Jones IDF is defined as:

IDF(qᵢ) = ln( (N - n(qᵢ) + 0.5) / (n(qᵢ) + 0.5) + 1 )

The constant 0.5 is a smoothing parameter that prevents division by zero and handles small document counts. The addition of 1 inside the natural logarithm prevents negative IDF values when a term appears in more than half of the collection’s documents. Terms that appear in nearly every document yield an IDF close to zero, while terms that appear in only a handful of documents yield high positive values.

The second component is the saturated term frequency. Letting f(qᵢ, D) represent the raw frequency of term qᵢ in document D, the basic saturation term takes the form:

TF_sat = f(qᵢ, D) / (f(qᵢ, D) + k₁)

Here, k₁ is a positive tuning parameter. When f(qᵢ, D) is very small, the denominator is dominated by k₁, causing the score to grow rapidly. When f(qᵢ, D) grows large, the f(qᵢ, D) term in the denominator dominates, forcing the fraction to approach a ceiling of 1.

The third component is the length normalization modifier, denoted as B. This modifier scales the k₁ parameter based on the ratio of document length |D| to average document length avgdl:

B = (1 - b) + b * (|D| / avgdl)

Here, b is a parameter between 0 and 1. If a document matches the average length exactly, |D| / avgdl equals 1, and B simplifies to 1. If the document is shorter than average, B is less than 1, reducing the effective k₁ and accelerating saturation. If the document is longer than average, B exceeds 1, increasing the denominator and penalizing the document.

Assembling these components yields the complete Okapi BM25 scoring equation for a multi-term query Q containing terms q₁ through qₘ against document D:

Score(D, Q) = Σ IDF(qᵢ) * [ f(qᵢ, D) * (k₁ + 1) ] / [ f(qᵢ, D) + k₁ * (1 - b + b * |D| / avgdl) ]

The term (k₁ + 1) in the numerator is an amplitude scaling constant. When f(qᵢ, D) is large, the fraction approaches (k₁ + 1), ensuring that the maximum possible term score scales directly with the chosen k₁ value.

What k1 actually controls

The k1 parameter controls term frequency saturation velocity. It dictates how fast the score curve bends as a query term is repeated within a document. Choosing a k1 value is an editorial decision about how much weight an information retrieval system should place on repeated keyword occurrences.

If k1 is set to 0, term frequency is completely removed from the calculation. Under k1 = 0, the term frequency fraction simplifies to 1 regardless of whether the term appears once or one thousand times. The algorithm functions as a purely binary retrieval model: either a word exists in the document or it does not.

As k1 increases, the ceiling of the saturation curve rises, allowing repeated mentions to continue contributing meaningful score increases. If k1 is set to an extremely high number like 100, the saturation effect is postponed, and BM25 behaves almost identically to classic linear TF-IDF. The table below demonstrates how the score of a document of average length changes across different k1 settings as term frequency rises.

Notice how k1 controls the difference between moderate repetition and heavy repetition. At k1 = 0.5, increasing term frequency from two to fifty mentions provides almost no score improvement. At k1 = 2.0, repeating the word fifty times doubles the score compared to mentioning it twice. In modern web search, where spammers actively repeat terms, lower k1 values provide stronger resistance against manipulation.

What b actually controls

The b parameter controls the intensity of document length normalization. It determines how strictly the search engine penalizes documents that exceed the average document length, and how generously it rewards concise pages. The parameter b is strictly constrained between 0.0 and 1.0.

When b = 0.0, document length normalization is completely disabled. The term b * (|D| / avgdl) drops out of the equation entirely, leaving the length modifier B locked at 1.0. Under this setting, a 10,000-word book and a 100-word summary are judged on raw term counts alone. Disabling length normalization harms search quality because long documents naturally accumulate higher word counts without being more focused.

When b = 1.0, document length normalization operates at maximum intensity. The denominator scales directly in proportion to document length. If Document A is four times longer than average, its effective k1 penalty is quadrupled. This setting assumes that all long documents are merely verbose versions of short documents, which can unfairly penalize legitimate, comprehensive manuals.

Testing different values of b allows engineers to adapt BM25 to specific document corpora. In a dictionary collection where every entry defines a single concept, b should be high. In an academic library where research papers naturally require extensive methodological descriptions, b should be lower.

Why 1.2 and 0.75 are the defaults

The search industry universal adoption of k1 = 1.2 and b = 0.75 as default values is not an arbitrary convention. These exact constants emerged from rigorous empirical experimentation during the Text REtrieval Conference (TREC) workshops conducted by the National Institute of Standards and Technology (NIST).

During TREC-3 and TREC-4 in the mid-1990s, the Okapi team evaluated dozens of parameter combinations across hundreds of thousands of news articles, congressional records, and technical abstracts. Researchers ran systematic grid sweeps, testing k1 values from 0.0 to 3.0 and b values from 0.0 to 1.0 against thousands of standardized search queries with human-judged relevance ground truth.

The grid searches revealed an optimal performance plateau centered at k1 between 1.2 and 1.6, and b between 0.70 and 0.80. Settings within this envelope consistently achieved the highest Mean Average Precision (MAP) and normalized discounted cumulative gain across diverse test collections. Setting k1 = 1.2 bends the saturation curve at the exact threshold where human readers perceive topical focus without rewarding keyword stuffing. Setting b = 0.75 dampens verbosity without eliminating comprehensive guides.

Because these defaults performed reliably across every major open benchmark, software libraries like Apache Lucene and Elasticsearch adopted them as standard configuration settings. Unless an organization possesses massive domain-specific click logs to train custom hyperparameters, altering k1 = 1.2 and b = 0.75 rarely improves retrieval quality.

Worked example end to end

To understand how BM25 scores documents in practice, let us walk through a complete numerical calculation on a toy corpus. Suppose our search engine has indexed a collection of five documents (N = 5) with the word counts and contents shown below.

• Document 1: 100 words, mentions “search” 3 times, mentions “index” 0 times.
• Document 2: 200 words, mentions “search” 1 time, mentions “index” 2 times.
• Document 3: 400 words, mentions “search” 2 times, mentions “index” 4 times.
• Document 4: 80 words, mentions “search” 0 times, mentions “index” 2 times.
• Document 5: 120 words, mentions “search” 0 times, mentions “index” 0 times.

Our user submits the multi-term query “search index”. We will score each document using standard BM25 parameters (k1 = 1.2, b = 0.75).

First, we calculate collection-level statistics. The total document count is N = 5. The total words across all documents is 100 + 200 + 400 + 80 + 120 = 900 words. The average document length is avgdl = 900 / 5 = 180 words.

Next, we calculate the inverse document frequency for each query term. The term “search” appears in three documents (Doc 1, Doc 2, Doc 3), so n(search) = 3. The term “index” appears in three documents (Doc 2, Doc 3, Doc 4), so n(index) = 3. We compute their IDF values:

IDF(search) = ln( (5 - 3 + 0.5) / (3 + 0.5) + 1 ) = ln( 2.5 / 3.5 + 1 ) = ln(1.714) ≈ 0.539
IDF(index)  = ln( (5 - 3 + 0.5) / (3 + 0.5) + 1 ) = ln( 2.5 / 3.5 + 1 ) = ln(1.714) ≈ 0.539

Now, we calculate the length normalization modifier B for each candidate document using B = (1 - 0.75) + 0.75 * (|D| / 180):

• Doc 1 (|D| = 100): B = 0.25 + 0.75 * (100 / 180) = 0.25 + 0.417 = 0.667
• Doc 2 (|D| = 200): B = 0.25 + 0.75 * (200 / 180) = 0.25 + 0.833 = 1.083
• Doc 3 (|D| = 400): B = 0.25 + 0.75 * (400 / 180) = 0.25 + 1.667 = 1.917
• Doc 4 (|D| = 80): B = 0.25 + 0.75 * (80 / 180) = 0.25 + 0.333 = 0.583

Next, we calculate the term score components for each document. The numerator multiplier is k1 + 1 = 2.2.

For Document 1 (f_search = 3, f_index = 0):

Score(search) = 0.539 * (3 * 2.2) / (3 + 1.2 * 0.667)
              = 0.539 * 6.6 / 3.80
              = 0.539 * 1.737 ≈ 0.936

Document 1 total score is 0.936 + 0 = 0.936.

For Document 2 (f_search = 1, f_index = 2):

Score(search) = 0.539 * (1 * 2.2) / (1 + 1.2 * 1.083)
              = 0.539 * 2.2 / 2.30
              = 0.539 * 0.957 ≈ 0.516

Score(index)  = 0.539 * (2 * 2.2) / (2 + 1.2 * 1.083)
              = 0.539 * 4.4 / 3.30
              = 0.539 * 1.333 ≈ 0.719

Document 2 total score is 0.516 + 0.719 = 1.235.

For Document 3 (f_search = 2, f_index = 4):

Score(search) = 0.539 * (2 * 2.2) / (2 + 1.2 * 1.917)
              = 0.539 * 4.4 / 4.30
              = 0.539 * 1.023 ≈ 0.551

Score(index)  = 0.539 * (4 * 2.2) / (4 + 1.2 * 1.917)
              = 0.539 * 8.8 / 6.30
              = 0.539 * 1.397 ≈ 0.753

Document 3 total score is 0.551 + 0.753 = 1.304.

For Document 4 (f_search = 0, f_index = 2):

Score(index)  = 0.539 * (2 * 2.2) / (2 + 1.2 * 0.583)
              = 0.539 * 4.4 / 2.70
              = 0.539 * 1.630 ≈ 0.879

Document 4 total score is 0 + 0.879 = 0.879.

Document 5 contains neither term, yielding a score of 0.000.

Notice how the ranking reflects both coverage and length. Document 3 ranks first because it matches both terms with healthy counts, though its length penalty keeps it close to Document 2. Document 1 beats Document 4 because its three mentions of “search” in a short 100-word document provide higher saturation than Document 4’s two mentions of “index”.

Where BM25 sits in a modern search engine

Modern web search engines do not rely on BM25 to produce their final public search results. Instead, BM25 operates as the foundational workhorse of candidate retrieval in a multi-stage ranking pipeline. When a user submits a query to Google or Bing, the engine cannot run deep transformer models across billions of web pages in fifty milliseconds.

Search engines solve this latency constraint by organizing ranking into distinct operational tiers. The first tier is candidate generation, often called first-pass retrieval. In this stage, inverted index shards scan their posting lists using BM25 or learned sparse retrieval algorithms.

BM25 filters the entire web down to the top one thousand candidate documents that share exact lexical tokens with the query. You can see how this fits into overall search architecture in our overview of how search engines rank pages.

Once BM25 generates the top one thousand candidates, the search pipeline passes those documents to second-pass and third-pass machine learning rerankers. These later stages evaluate dense semantic embeddings, cross-encoder neural networks, BERT models, query intent signals, and link graph authority. BM25 provides the speed required to filter billions of documents, while neural networks provide the semantic understanding required to rank the finalists.

BM25 also remains the backbone of enterprise search platforms. Databases like Elasticsearch, OpenSearch, Solr, and Postgres full-text search utilize BM25 as their default relevance engine. For internal documentation, code repositories, and ecommerce catalogs, BM25 provides immediate, predictable relevance without requiring costly GPU inference clusters.

What this means for writing content

Understanding the mathematical mechanics of BM25 directly informs effective content creation and search optimization. Because search engines evaluate text through term saturation and length normalization, outdated tactics that attempt to manipulate word counts or repetition backfire.

First, keyword stuffing is mathematically useless. Repeating a target phrase fifteen times across an article provides virtually no score increase over mentioning it four times naturally in relevant subheadings. Once the saturation curve reaches its asymptote, additional mentions add fractions of a point while degrading human readability. Search engines also employ spam filters that detect unnatural term distributions and penalize documents that deviate from natural linguistic patterns.

Second, document length is not an automatic ranking lever. For years, digital marketing blogs claimed that publishing 3,000-word ultimate guides guaranteed higher rankings than publishing 800-word answers. BM25 proves the opposite: increasing word count without increasing topical substance actively harms your relevance score by triggering length normalization penalties. If you add 1,500 words of filler text to an article, you dilute its term density and lower its BM25 score.

Third, write with comprehensive lexical breadth rather than narrow repetition. BM25 scores multi-term queries by summing the scores of each distinct query term. An article that uses related technical entities, synonyms, and natural subtopics matches more terms in complex user queries than an article that repeats the same primary phrase. You can learn how to structure content around mechanisms rather than keywords in our foundational guide on what SEO is.

Frequently asked questions

What does BM25 stand for?

BM25 stands for Best Matching 25. It represents the twenty-fifth iteration of the probabilistic ranking functions developed during the Okapi information retrieval experiments in the 1990s. Computer scientists Stephen Robertson, Karen Sparck Jones, and their colleagues created it at City University London to refine probabilistic relevance weighting for full-text document collections.

Is BM25 better than TF-IDF?

BM25 outperforms standard TF-IDF across almost all practical text retrieval benchmarks. Classic TF-IDF increases relevance scores linearly with term frequency and fails to account for varying document lengths accurately. BM25 introduces asymptotic term frequency saturation and robust document length normalization. These mathematical improvements prevent keyword-stuffed documents and arbitrarily long articles from dominating search results.

Does Google use BM25?

Google does not rely exclusively on vanilla BM25, but its core candidate retrieval systems use closely related probabilistic lexical matching principles. Modern web search engines utilize BM25 variants and learned sparse retrieval models during first-pass retrieval to extract candidate documents from inverted index shards. Later ranking phases apply machine learning rerankers, semantic embeddings, and link graph algorithms.

What are good values for k1 and b?

Standard information retrieval practice sets k1 to 1.2 and b to 0.75. These default parameters were established during extensive TREC benchmark evaluations and perform well across general document corpora. Lowering k1 toward 0.5 dampens the effect of term frequency, while increasing b toward 1.0 penalizes long documents more severely.

What is term frequency saturation?

Term frequency saturation is a mathematical property where each additional occurrence of a query term provides diminishing marginal relevance score increases. Under BM25, encountering a word for the first time yields the largest score increase. Repeating that same word fifty additional times approaches a strict mathematical ceiling, preventing keyword spam from inflating document scores.

Is BM25 still relevant with AI search?

BM25 remains essential in modern AI search architectures, where it powers hybrid retrieval systems alongside dense vector embeddings. Dense retrieval handles conceptual synonyms and semantic intent, while BM25 provides precise lexical matching for exact entity names, model numbers, and rare terminology. Combining BM25 with neural reranking delivers faster candidate retrieval than vector search alone.

What is the difference between BM25 and BM25F?

BM25 treats every document as a single unstructured block of text. In contrast, BM25F extends the ranking function to handle structured fields such as titles, headings, body text, and anchor text. BM25F calculates weighted term frequencies across distinct document zones before applying saturation, giving higher relevance to words found in titles.

Can I calculate BM25 by hand?

You can calculate BM25 by hand for small text collections using basic arithmetic. The formula requires counting term frequencies in the document, determining document length relative to average corpus length, and computing the inverse document frequency. While manual calculations take only minutes for single terms, search engines compute millions of these equations per second.