How does BM25 work?
- Authors
- Name
- Amit Shekhar
- Published on
BM25 is a scoring formula that search engines use to rank documents for a search query. It gives a higher score to a document when it contains the query words many times, when those words are rare across all documents, and when the document is not unnecessarily long.
In this blog, we will learn about how BM25 works. We will also see why simple word counting fails, how BM25 rewards rare words using IDF, how it limits a repeated word using saturation, how it treats long and short documents fairly using length normalization, a complete worked example with Python code, and where BM25 works well and where it fails.
We will cover the following:
- The problem: ranking documents for a query
- The simple approach: counting words
- What is BM25?
- Idea 1: Rare words matter more (IDF)
- Idea 2: Repeating a word has a limit (saturation)
- Idea 3: Long documents must not win unfairly (length normalization)
- The full BM25 formula
- A complete worked example
- BM25 in Python code
- What do k1 and b control?
- TF-IDF vs BM25
- Where BM25 works well and where it fails
I am Amit Shekhar, Founder @ Outcome School, I have taught and mentored many developers, and their efforts landed them high-paying tech jobs, helped many tech companies in solving their unique problems, and created many open-source libraries being used by top companies. I am passionate about sharing knowledge through open-source, blogs, and videos.
I teach AI and Machine Learning at Outcome School.
Let's get started.
The problem: ranking documents for a query
Before jumping into BM25, we must understand the problem it solves.
Let's say we have a library with one million articles. Someone types "cat food" in the search box. Many articles contain these words. Some are exactly about cat food. Some just mention a cat once. Some are about dog food.
We must show the best ones on top. So, we need a way to give every article a score that tells how well it matches the search. Then, we sort by this score.
This is called ranking. BM25 is one of the most famous ways of doing ranking. It is used inside search tools like Elasticsearch, and it is also used in RAG systems to find the right text to give to an LLM.
The simple approach: counting words
The most natural idea is: count how many times the query terms appear in each document. More count, higher score.
This count is called the term frequency. If "cat" appears 3 times in a document, the term frequency of "cat" in that document is 3.
It looks reasonable, but the following issues arise:
- Problem 1: Common words win. Suppose the query is "the cat". The word "the" appears in almost every document many times. So, documents full of "the" will win, even if they are not about cats.
- Problem 2: Repetition wins. A spammy page that writes "cat cat cat cat cat" 100 times will beat a genuine page about cats.
- Problem 3: Long documents win. A 50-page book naturally contains "cat" more times than a 1-paragraph article, even if the article is more focused on cats.
So, here comes BM25 to the rescue. It fixes each of these three problems with one idea each.
What is BM25?
BM25 is a ranking formula that scores how relevant a document is to a query, based on how often the query terms appear in the document, how rare those terms are in the whole collection, and how long the document is.
BM stands for Best Matching, and 25 is simply the version number. It came from the work of Stephen Robertson, Karen Spärck Jones, and others in the Okapi search system in the 1990s. That is why it is also called Okapi BM25.
The important point is that BM25 does not understand meaning. It only looks at words and counts. Search by meaning is a different method, and we have a detailed blog on Semantic Search that explains how it works.
Let's understand the three ideas one by one.
Idea 1: Rare words matter more (IDF)
If we search "the cat", which word tells us more about what we want? Clearly "cat". The word "the" is everywhere, so it tells us almost nothing.
Rare words carry more information than common words.
BM25 measures how rare a term is using IDF, which stands for Inverse Document Frequency.
- Document frequency is the number of documents that contain the term.
- Inverse means we flip it. Fewer documents with the term means a higher IDF.
BM25 commonly uses this IDF formula:
IDF(term) = ln( (N - n + 0.5) / (n + 0.5) + 1 )
N = total number of documents
n = number of documents that contain the term
ln = natural logarithm
Here, let's see the behavior:
- If a term appears in very few documents,
nis small, so the IDF is big. - If a term appears in almost all documents,
nis close toN, so the IDF is close to zero.
The ln (logarithm) just keeps the numbers from becoming too big. The 0.5 values are small adjustments that keep the formula safe when n is 0. The + 1 inside makes sure the IDF never goes below zero.
Let's say we have 1000 documents:
"the" appears in 990 documents -> IDF = 0.01 (almost useless)
"cat" appears in 20 documents -> IDF = 3.89 (very useful)
Problem 1 is solved. Common words now count for almost nothing.
Idea 2: Repeating a word has a limit (saturation)
Now, let's fix the spam problem.
If a document mentions "cat" once, it is about cats a little. If it mentions "cat" 5 times, it is clearly about cats. But if it mentions "cat" 100 times, is it 20 times more about cats than the one with 5? No. After some point, more repetition tells us nothing new.
Let's take a real-world example. The first slice of pizza makes us very happy. The second makes us happy. By the tenth slice, one more slice adds almost no happiness. This is called saturation.
BM25 uses this idea. The score for term frequency grows quickly at first and then flattens. It uses a setting called k1 to control how fast it flattens. A common value is k1 = 1.2.
Ignoring document length for now, the term frequency part looks like below:
TF part = f * (k1 + 1) / (f + k1)
f = how many times the term appears in the document
Let's see the numbers with k1 = 1.2:
f = 1 -> 1.00
f = 2 -> 1.38
f = 3 -> 1.57
f = 5 -> 1.77
f = 10 -> 1.96
f = 100 -> 2.17
Here, we can see that going from 1 to 2 adds a lot, but going from 10 to 100 adds very little. The value can never cross k1 + 1, which is 2.2 here.
Problem 2 is solved. Writing a word 100 times does not help much.
Planning your move into AI?
No matter which tech domain you work in today, with your effort, you can transition to these roles:
- AI Engineer
- Forward Deployed Engineer
- AI Agent Engineer
- Harness Engineer
- Voice AI Engineer
- AI Architect
- LLM Engineer
- LLM Inference Engineer
- Gen AI Engineer
- MLOps Engineer
- ML Engineer
- Deep Learning Engineer
Our AI and Machine Learning Program at Outcome School takes you there.
Idea 3: Long documents must not win unfairly (length normalization)
Now, the last problem: long documents.
Suppose two documents both mention "cat" 2 times. One is 10 words long. The other is 1000 words long. In the short one, cats are a big part of the text. In the long one, cats are just a small mention.
The same count means more in a short document than in a long document.
BM25 handles this by comparing the document's length with the average document length in the collection. This adjustment is called length normalization.
It uses another setting called b, which is a number between 0 and 1. A common value is b = 0.75.
length factor = 1 - b + b * (document length / average length)
Let's see what this does:
- If the document has exactly the average length, the factor is 1. No change.
- If the document is longer than average, the factor is bigger than 1, which lowers the score.
- If the document is shorter than average, the factor is smaller than 1, which raises the score.
The setting b controls how strong this effect is. With b = 0, length is ignored completely. With b = 1, length has the full effect.
Problem 3 is solved.
The full BM25 formula
Now, we have all three ideas. Let's put them together.
For each term in the query, BM25 calculates:
term score = IDF(term) * ( f * (k1 + 1) ) / ( f + k1 * (1 - b + b * |D| / avgdl) )
f = how many times the term appears in document D
|D| = length of document D (number of words)
avgdl = average document length in the collection
k1 = saturation setting (commonly 1.2)
b = length setting (commonly 0.75)
Then, it adds the term scores for all the query terms:
BM25(document, query) = sum of term scores for every term in the query
Here, we can notice that the length factor sits inside the denominator, next to k1. So, a long document makes the score flatten faster. It needs more repetitions to get the same score as a short document.
So, for each query word, BM25 combines three things:
term score = how rare the word is * how often it appears (with a limit and an adjustment for length)
A quick note for you
No matter which tech domain you work in, get familiar with these topics:
- LLM
- RAG
- MCP
- Agent
- Fine-tuning
- Quantization
We put it all together in one video:
AI Engineering Explained: LLM, RAG, MCP, Agent, Fine-Tuning, and Quantization
No need to stop reading - bookmark it and watch later when you get time. Future you will thank you.
Now, let's get back to the topic.
A complete worked example
The best way to learn this is by taking an example.
Let's say our collection has only 4 documents:
D1: "the cat eats cat food" (5 words)
D2: "dog food is cheap" (4 words)
D3: "the cat sat on the mat near the door" (9 words)
D4: "fresh food for every pet" (5 words)
The query is "cat food". We will use k1 = 1.2 and b = 0.75.
Step 1: We find the basics.
N = 4 documents
avgdl = (5 + 4 + 9 + 5) / 4 = 5.75 words
Step 2: We find the IDF of each query term.
"cat" appears in D1 and D3, so n = 2. "food" appears in D1, D2, and D4, so n = 3.
IDF(cat) = ln( (4 - 2 + 0.5) / (2 + 0.5) + 1 ) = ln(2.00) = 0.69
IDF(food) = ln( (4 - 3 + 0.5) / (3 + 0.5) + 1 ) = ln(1.43) = 0.36
Here, we can see that "cat" is rarer, so it gets almost double the weight of "food".
Step 3: We find the TF part for each document and term.
D1: cat f=2 -> 1.43, food f=1 -> 1.06
D2: cat f=0 -> 0, food f=1 -> 1.14
D3: cat f=1 -> 0.81, food f=0 -> 0
D4: cat f=0 -> 0, food f=1 -> 1.06
Here, we can notice something interesting. "food" appears once in D2, D4, and D1. But D2 gets 1.14, while D1 and D4 get 1.06. Why? Because D2 is shorter (4 words) than D1 and D4 (5 words). The length normalization is working.
Similarly, "cat" appears once in D3, but D3 gets only 0.81 because it is a long document (9 words).
Step 4: We multiply by IDF and add up.
D1: 0.69 x 1.43 + 0.36 x 1.06 = 0.99 + 0.38 = 1.37
D2: 0.36 x 1.14 = 0.41
D3: 0.69 x 0.81 = 0.56
D4: 0.36 x 1.06 = 0.38
Step 5: We rank the documents.
1st: D1 (1.37) "the cat eats cat food"
2nd: D3 (0.56) "the cat sat on the mat near the door"
3rd: D2 (0.41) "dog food is cheap"
4th: D4 (0.38) "fresh food for every pet"
This makes sense. D1 has both words, so it wins clearly. D3 comes second because "cat" is the rarer and more important word. D2 beats D4 only because it is shorter.
This is how BM25 ranks documents.
BM25 in Python code
Now, let's see the code for the same example. We can write the code as below:
import math
docs = [
"the cat eats cat food",
"dog food is cheap",
"the cat sat on the mat near the door",
"fresh food for every pet",
]
query = "cat food"
k1, b = 1.2, 0.75
tokenized = [d.split() for d in docs]
N = len(tokenized)
avgdl = sum(len(d) for d in tokenized) / N
def idf(term):
n = sum(1 for d in tokenized if term in d)
return math.log((N - n + 0.5) / (n + 0.5) + 1)
def bm25(doc):
score = 0.0
for term in query.split():
f = doc.count(term)
tf_part = f * (k1 + 1) / (f + k1 * (1 - b + b * len(doc) / avgdl))
score += idf(term) * tf_part
return score
for i, doc in enumerate(tokenized, 1):
print(f"D{i}: {bm25(doc):.2f}")
Here, we have:
tokenizedsplits every document into a list of words.Nis the number of documents andavgdlis the average length.idf(term)counts how many documents contain the term and applies the IDF formula.bm25(doc)loops over the query terms, counts each term in the document, applies the saturation and length factor, multiplies by the IDF, and adds everything.
It will print the following:
D1: 1.37
D2: 0.41
D3: 0.56
D4: 0.38
It works perfectly, and it matches our hand calculation.
Note: Real search engines do not loop over every document like this. They build an inverted index first. An inverted index is like the index at the back of a book. For every word, it stores the list of documents that contain it, along with the counts. So, for "cat food", the engine looks at only the documents in the "cat" list and the "food" list. That is why BM25 search is very fast, even on millions of documents.
Stay updated: Subscribe to our newsletter to get our latest AI and Machine Learning blogs straight to your inbox.
What do k1 and b control?
Let me summarize the two settings:
- k1 controls saturation. A small
k1means repetition stops helping very quickly. A largek1means repetition keeps helping for longer. Withk1 = 0, only the presence of the word matters, not the count. Common values are between 1.2 and 2.0. - b controls length normalization. With
b = 0, length is ignored. Withb = 1, length matters fully. The common value is 0.75.
Most of the time, the default values work well. We change them only when we test on our own data and see a clear improvement.
TF-IDF vs BM25
BM25 is often compared with an older method called TF-IDF, which simply multiplies term frequency by IDF.
Let me tabulate the differences between TF-IDF and BM25 for your better understanding.
| Point | TF-IDF | BM25 |
|---|---|---|
| Rewards rare words | Yes, with IDF | Yes, with IDF |
| Effect of repeating a word | Keeps growing (often softened with a log, but no hard limit) | Flattens towards a fixed limit (saturation) |
| Handles document length | Usually through a separate normalization | Built in, controlled by b |
| Settings to tune | Almost none | k1 and b |
| Ranking quality | Good | Usually better, a common default in search engines |
Where BM25 works well and where it fails
Advantages:
- It is fast, even on millions of documents, because of the inverted index.
- It needs no training and no GPU.
- It is great at exact matches, like product codes, error messages, names, and rare technical words.
- It is easy to explain why a document ranked high.
Disadvantages:
- It does not understand meaning. A search for "car" will not match a document that only says "automobile".
- It ignores word order. "dog bites man" and "man bites dog" look the same to it.
- It does not understand that "not good" is different from "good".
This is why modern search systems often use hybrid search. They run BM25 for exact word matching and vector search (which finds text with similar meaning using embeddings) together, and then combine both results. BM25 catches the exact words, vector search catches the meaning.
Now, we must have understood how BM25 works, how it scores documents, and where it works well and where it fails.
To master RAG, Vector Databases, and Embeddings, and to build an AI Tutor from scratch, check out our AI and Machine Learning Program at Outcome School.
Frequently Asked Questions
What does BM in BM25 stand for?
BM stands for Best Matching, and 25 is simply the version number. BM25 came from the work of Stephen Robertson, Karen Spärck Jones, and others in the Okapi search system in the 1990s. That is why it is also called Okapi BM25.
Does BM25 need training or a GPU?
No. BM25 needs no training and no GPU. It only looks at words and counts, using a fixed formula with two settings, k1 and b. Most of the time, the default values of these settings work well, and we change them only when we test on our own data and see a clear improvement.
Why is BM25 search fast on millions of documents?
BM25 search is fast because real search engines build an inverted index first. For every word, the inverted index stores the list of documents that contain it, along with the counts. So, the engine looks at only the documents in the lists of the query words, instead of looping over every document.
Can BM25 find documents that use synonyms?
No. BM25 does not understand meaning, so a search for "car" will not match a document that only says "automobile". This is why modern search systems often combine BM25 with vector search, which finds text with similar meaning using embeddings. BM25 catches the exact words, and vector search catches the meaning.
That's it for now.
Thanks
Amit Shekhar
Founder @ Outcome School
You can connect with me on:
Follow Outcome School on:
