BM25 is a ranking function that scores how well a document matches a keyword query. It looks at how often each query word appears in the document, how rare that word is across the whole collection, and how long the document is. Search engines, site search tools and many AI retrieval systems use it to produce a first list of candidate pages or passages.
The name stands for Best Matching 25. It comes from a line of research on probabilistic information retrieval and was made popular by the Okapi search system, which is why it is sometimes called Okapi BM25. For years it has been the default keyword ranking method in widely used open source search software such as Lucene, Elasticsearch and OpenSearch.
How does BM25 score a document?
BM25 treats a query as a set of words, called terms, and gives each document a score by adding up a contribution for every query term it contains. Word order and meaning play no part. Three factors decide how large each contribution is.
Term frequency with saturation
Term frequency is the number of times a query term appears in the document. More occurrences raise the score, but each extra occurrence adds less than the one before. A page that mentions "invoice" ten times scores higher than one that mentions it once, yet only a little higher than one that mentions it five times. This saturation stops a page from winning simply by repeating a word.
Inverse document frequency
Inverse document frequency (IDF) measures how rare a term is across all documents in the index. A word that appears in almost every document, such as "the" or "page", tells the system little and gets a weight close to zero. Words found in only a few documents, such as a product name or a technical term, get a high weight, so a match on a rare query term counts for much more than a match on a common one.
Document length normalization
Long documents contain more words, so they match more terms by chance. BM25 compares each document's length with the average length in the collection and lowers the score of documents that are longer than average. As a result, a short passage that mentions a term twice can outscore a long article that mentions it three times.
What does the BM25 formula look like?
For a query Q and a document D, the score is a sum over each query term q:
score(D, Q) = sum of IDF(q) * f(q, D) * (k1 + 1) / (f(q, D) + k1 * (1 - b + b * |D| / avgdl))| Symbol | Meaning | ||
|---|---|---|---|
| f(q, D) | How many times term q appears in document D | ||
| IDF(q) | Weight based on how few documents contain q | ||
| \ | D\ | Length of document D in words | |
| avgdl | Average document length in the collection | ||
| k1 | Controls how quickly term frequency saturates | ||
| b | Controls how strongly length is penalized, from 0 (no penalty) to 1 (full penalty) |
The two parameters are tuned per collection. Common defaults put k1 around 1.2 and b at 0.75, and many systems never change them. Because the formula needs only word counts, it runs fast over very large indexes using an inverted index, a lookup table that lists which documents contain each word.
Why is BM25 still used in AI search?
AI search systems retrieve text before a language model writes an answer, a pattern called retrieval-augmented generation. Retrieval often starts with BM25 because it is cheap and predictable and needs no training. It is especially good at exact matches: product codes, error messages, names, version numbers and rare jargon.
Modern systems usually pair it with vector search, which compares embeddings, lists of numbers that encode meaning. The two methods find different results, so many pipelines run both and merge the ranked lists, an approach called hybrid search. A later reranking step then scores the merged candidates more carefully before passages reach the model. How AI search chooses which sources to cite describes those later stages.
What are the limits of BM25?
BM25 only sees the words that are on the page. If a user searches for "cheap laptop" and a page says "affordable notebook", the page scores zero for those terms, a problem known as vocabulary mismatch. Nor can it tell that "bank" in a finance article and "bank" in a fishing article mean different things.
| BM25 | Vector search | |
|---|---|---|
| Matches on | Exact words | Meaning |
| Synonyms and paraphrases | Missed | Usually found |
| Rare names and codes | Strong | Can be weak |
| Needs a trained model | No | Yes |
| Easy to explain a score | Yes | Harder |
These gaps made semantic search common, while the strengths listed above explain why BM25 stayed in use alongside it.
What does BM25 mean for a website?
When a retrieval system uses BM25, a page or passage can only be found through it if it contains the words people actually use in their questions. A page that describes a product only in brand language, without the plain terms a buyer would type, gives BM25 nothing to match. Using the common name of a thing at least once, near the text that explains it, keeps the passage eligible.
Repetition does not help much, because term frequency saturates and length normalization penalizes padding. Rare, specific terms carry the most weight, so precise product names, model numbers and technical vocabulary make a passage easier to retrieve for the queries that contain them. Short, focused sections also benefit, since a passage that covers one topic is not diluted by the length of unrelated text around it.
