BM25
BM25 is a ranking function that scores a document against a query using term frequency and inverse document frequency, with two corrections TF-IDF lacks: term frequency saturates rather than growing without limit, and scores are normalised for document length.
BM25 — the name is short for “Best Matching 25”, from a series of experiments — is the standard lexical ranking function. It is the default in Elasticsearch and Lucene, and it remains a strong baseline that newer neural methods are measured against rather than having simply replaced.
The two corrections it makes to TF-IDF are the reason it works better:
Saturation. The tenth occurrence of a term adds far less than the second. A
tunable parameter, usually written k1, controls how quickly the curve flattens.
This removes the incentive for pure repetition.
Length normalisation. A long document naturally contains more occurrences of
everything, so raw term frequency favours length. BM25 divides by document
length relative to the collection average, with a parameter b controlling how
strongly. Setting b to zero disables it entirely.
For a search engine you build yourself, BM25 is the point at which results start feeling genuinely useful. It handles lexical matching well and understands nothing about meaning — which is exactly why modern engines combine it with learned representations rather than choosing one or the other.