BM25 vs Dense Retrieval: Measure It on Your Own Corpus

1. What you'll build: a five-arm BM25 vs dense retrieval bake-off

# 50 candidates retrieved per arm, reranked, and scored at K=10arm                    Recall@10    MRR@10   NDCG@10    ms/query      $/1k qBM25                      0.7739    0.6312    0.6617         0.4     0.00001BM25 + stemmer            0.8187    0.6492    0.6863         0.6     0.00001Dense                     0.7833    0.6047    0.6451        24.3     0.00034Hybrid (RRF)              0.8352    0.6571    0.6962        24.8     0.00034Hybrid + rerank           0.8239    0.6608    0.6931      9024.2     0.12534

That is the harness you are about to build. Each row is an arm: one retrieval strategy, scored against 300 queries that carry real human relevance judgments. Watch NDCG@10. It is a 0-to-1 score for how high the relevant documents landed in the top ten. On this corpus 0.02 is a large move, and anything under 0.001 is score-tie noise that section 5 explains how to remove. The interesting deltas below sit between those two bounds, which is exactly why step 10 has to decompose them rather than read them off the table. Section 3 unpacks all three metrics.

One tokenizer flag - a Snowball stemmer, off by default - bought +0.0246 NDCG@10 for well under a millisecond per query. That leaves it 0.0068 NDCG@10 below the hybrid-plus-cross-encoder row, which costs four orders of magnitude more per query, and 0.0099 below the best row in the table. I only found it because I checked our BM25 number against the bm25s author's published ablation table and realised I had been running the unstemmed configuration the whole time.

The cross-encoder made retrieval worse. Against plain reciprocal rank fusion (RRF) it lost 0.0113 Recall@10 and 0.0031 NDCG@10 while gaining 0.0037 MRR@10, and the last step of this tutorial shows you exactly where that gain came from.

The rule of thumb everyone repeats - BM25 wins on rare exact tokens, dense wins on paraphrases - does not survive this corpus. Rare gene and species names sit on both sides of the disagreement, and 167 of the 300 queries score identically on both arms.

By the end you will have a bake-off harness that runs five retrieval strategies over your own labelled query set and reports Recall@k, MRR@k, NDCG@k, latency and cost per query for each, so you can stop guessing which retriever your corpus needs. It is an intermediate build: you should have shipped a RAG pipeline before and know what an embedding and a top-k are. Budget about 90 minutes on a fast machine and closer to two and a half hours on a slow one. Most of that is waiting: a large torch install, then roughly 45 minutes of cross-encoder time, then the corpus encode, which runs three times unless you skip the diagnostic in step 5.

Verified against Python 3.13.9, bm25s==0.3.10, sentence-transformers==5.7.0, numpy==2.5.1, ir_datasets==0.6.3, pytest==9.1.1, PyStemmer==3.1.0, and torch==2.13.0 and transformers==5.14.1 as resolved by sentence-transformers, on 2026-08-10. Every code block on this page was executed end to end in a scratch project built from nothing, including the full 15,000-pair cross-encoder pass. Everything runs locally on CPU. No API keys anywhere.

This is the hands-on companion to the RAG Engineering in Production series. Part 1 argues that the retrieval strategy you default to is usually wrong for your data. This tutorial is the code that lets you check.

2. Prerequisites: Python 3.13, bm25s, sentence-transformers

Everything here was run on Python 3.13.9. Python 3.12 is the floor, and the pins set it: numpy==2.5.1 declares Requires-Python: >=3.12, and it is the strictest of the seven. Budget about 2 GB of free disk for the weights and the corpus. No GPU, no accounts, no credentials - the Hugging Face downloads here are anonymous.

Create the project directory and a virtual environment:

mkdir bakeoffcd bakeoffpython -m venv .venv

Write the dependency list.

File: requirements.txt

bm25s==0.3.10sentence-transformers==5.7.0numpy==2.5.1ir_datasets==0.6.3pytest==9.1.1PyStemmer==3.1.0

torch and transformers are not listed: sentence-transformers pulls both, and pinning them here as well risks a conflict with whatever resolution it asks for. The table below records the versions they resolved to, so you can check yours against them.

Install it. Windows first, then macOS and Linux:

# Windows.venv/Scripts/python.exe -m pip install -r requirements.txt

# macOS and Linux.venv/bin/python -m pip install -r requirements.txt

sentence-transformers pulls torch, several hundred megabytes of it. It took between 8 and 12 minutes on our runs, even with a warm pip cache, so start it and go make coffee. Every command from here on uses one of those two interpreter paths. Pick yours once and substitute it throughout - the command always ends in the script name, so only the prefix changes.

These are the versions the numbers in this tutorial came from:

Now prove the install works before you write anything else.

File: check_env.py

# check_env.pyimport bm25simport numpy as npfrom sentence_transformers import SentenceTransformerprint("bm25s        ", bm25s.__version__)print("numpy        ", np.__version__)model = SentenceTransformer("sentence-transformers/all-MiniLM-L6-v2")print("embedding dim", model.get_sentence_embedding_dimension())

Run it from the project root:

.venv/Scripts/python.exe check_env.py

The first run downloads all-MiniLM-L6-v2, about 90 MB, with no progress output for the first few seconds. Expected stdout:

bm25s         0.3.10numpy         2.5.1embedding dim 384

On sentence-transformers 5.7.0 you will also see a FutureWarning about get_sentence_embedding_dimension on stderr, and a Hugging Face notice about unauthenticated requests. Both are harmless; section 5 (Troubleshooting) lists them verbatim.

3. How lexical, dense, fusion and joint scoring differ

A retriever is a function that decides what "relevant" means. The five arms in the table are built on four different answers to that question, and they fail in different places.

You know the first two. Lexical matching weights query terms by corpus rarity and document length, and costs nothing: no model, no embeddings, sub-millisecond queries. Vector similarity ranks by cosine distance in a shared embedding space, catches paraphrases with zero shared words, and charges you an encoder call per query. The usual gloss is that only the vector arm survives a paraphrase. Hold that thought until step 5, where BM25 scores zero on Admpchordin and the encoder finds it.

Rank fusion is where the interesting decision hides. Combining two retrievers by score means putting BM25's unbounded, corpus-dependent numbers on the same scale as cosine values between -1 and 1, then picking a weight for the sum. That weight is a hyperparameter, and you retune it every time the corpus shifts. Elasticsearch's own hybrid guidance shows the weighted-sum form alongside RRF, and the retuning bill is not the part that gets discussed. Reciprocal rank fusion reads only positions, so there is no scale to reconcile and no weight to choose.

Joint scoring feeds the query and one candidate through a single model together, which is what a cross-encoder does. In principle it is the most accurate of the four; in practice it runs once per candidate where the others run once per query, and step 9's latency column shows what that costs.

None of the four means anything without ground truth. Without a set of queries and human judgments of which documents answer them, you cannot compare any of these. Everything above stays an opinion until something scores it.

Three metrics turn a ranked list plus judgments into a number, and they answer different questions:

They disagree on purpose. Which one you optimise is a product decision. If your RAG pipeline feeds the top 10 chunks into a large context window, Recall@10 is your metric and rank order barely matters. If you show the user one answer, MRR is your metric.

4. Building the bake-off harness step by step

Step 1: Load BEIR SciFact with its relevance judgments in Python

Goal. Load BEIR's SciFact split as documents, queries, and qrels (query relevance judgments: for each query, which document ids a human marked relevant, and at what grade).

Why this step. A bake-off without ground truth is a vibe check. BEIR is a benchmark suite of retrieval datasets that ship published baselines, and SciFact is one of them: about five thousand scientific abstracts with real human relevance judgments attached, small enough to index on a laptop and honest enough to argue with. ir_datasets downloads and caches it in one call. Starting from real qrels means every later step has something to be right or wrong about.

Code.

File: corpus.py

# corpus.py"""SciFact from BEIR: ~5k scientific abstracts with real relevance judgments.Chosen because it ships qrels. A bake-off without ground truth is a vibe check."""import ir_datasets_ds = ir_datasets.load("beir/scifact/test")DOCS = [{"id": d.doc_id, "text": f"{d.title} {d.text}".strip()} for d in _ds.docs_iter()]QUERIES = [{"id": q.query_id, "text": q.text} for q in _ds.queries_iter()]QRELS: dict[str, dict[str, int]] = {}for qrel in _ds.qrels_iter():    QRELS.setdefault(qrel.query_id, {})[qrel.doc_id] = qrel.relevance# Keep only queries that actually have judgments; an unjudged query silently# scores zero on every arm and drags all four means down equally.QUERIES = [q for q in QUERIES if QRELS.get(q["id"])]if __name__ == "__main__":    print(f"documents        {len(DOCS)}")    print(f"judged queries   {len(QUERIES)}")    print(f"relevance pairs  {sum(len(v) for v in QRELS.values())}")

Run it.

.venv/Scripts/python.exe corpus.py

Expected output. The first run downloads the BEIR zip, which took about 63 seconds here, and prints ir_datasets progress lines to stderr before these three:

documents        5183judged queries   300relevance pairs  339

Now check the judgments are not degenerate.

File: check_qrels.py

# check_qrels.py"""How many relevant documents does a typical query have?A median of 1 changes how you read the final table: Recall@10 becomes close toa yes/no per query, and NDCG@10 has little grading left to do that MRR is notalready doing."""import statisticsfrom corpus import QRELScounts = [len(judged) for judged in QRELS.values()]print("min", min(counts), "median", statistics.median(counts), "max", max(counts))

.venv/Scripts/python.exe check_qrels.py

Expected output:

min 1 median 1.0 max 5

ir_datasets prints an [INFO] Opening ... line above that, naming the cache file inside your own home directory. Ignore it.

What just happened. Three module-level names now exist that every arm imports: DOCS, QUERIES, and QRELS. Loading happens at import time, so importing corpus anywhere gives the same 5,183 documents in the same order - which matters later, because embedding row i has to correspond to DOCS[i].

The median of 1 is the important finding. Most SciFact queries have exactly one relevant document, and all 339 judgments are binary. That shapes everything downstream: Recall@10 becomes close to a yes/no per query, and NDCG@10 has very little grading left to do that MRR is not already doing. Keep it in mind when you read the final table.

Step 2: Implement Recall@k, MRR@k and NDCG@k from scratch

Goal. Implement recall_at_k, mrr_at_k and ndcg_at_k, then prove they behave.

Why this step. You could import an evaluation library. Do not, at least not the first time. These are forty lines, and the whole point of the exercise is to know exactly what each number means when two of them disagree. Hand-writing them also removes a class of silent error: pytrec_eval's recip_rank applies no cutoff at all, so an "MRR@10" built on it credits a hit at rank 40 and looks fine until you compare it to a published baseline.

Code.

File: measure.py

# measure.py"""Recall@k, MRR@k and NDCG@k, hand-written so nothing is hidden.All three take (ranked doc_ids, {doc_id: grade}, k) and return a float."""import mathdef recall_at_k(ranked: list[str], relevant: dict[str, int], k: int) -> float:    if not relevant:        return 0.0    found = sum(1 for doc_id in ranked[:k] if relevant.get(doc_id, 0) > 0)    return found / len(relevant)def mrr_at_k(ranked: list[str], relevant: dict[str, int], k: int) -> float:    for rank, doc_id in enumerate(ranked[:k], start=1):        if relevant.get(doc_id, 0) > 0:            return 1.0 / rank    return 0.0def ndcg_at_k(ranked: list[str], relevant: dict[str, int], k: int) -> float:    dcg = sum(        (2 ** relevant.get(doc_id, 0) - 1) / math.log2(rank + 1)        for rank, doc_id in enumerate(ranked[:k], start=1)    )    ideal = sum(        (2 ** grade - 1) / math.log2(rank + 1)        for rank, grade in enumerate(sorted(relevant.values(), reverse=True)[:k], start=1)    )    return dcg / ideal if ideal else 0.0

File: test_measure.py

# test_measure.pyfrom measure import recall_at_k, mrr_at_k, ndcg_at_kRANKED = ["a", "b", "c", "d"]RELEVANT = {"c": 1, "z": 1}          # one found at rank 3, one never retrieveddef test_recall_counts_only_what_was_retrieved():    assert recall_at_k(RANKED, RELEVANT, 4) == 0.5def test_mrr_uses_the_first_hit():    assert mrr_at_k(RANKED, RELEVANT, 4) == 1 / 3def test_mrr_is_zero_when_the_hit_is_below_k():    assert mrr_at_k(RANKED, RELEVANT, 2) == 0.0def test_ndcg_is_one_for_a_perfect_ranking():    assert ndcg_at_k(["c"], {"c": 1}, 10) == 1.0def test_empty_relevant_does_not_divide_by_zero():    assert recall_at_k(RANKED, {}, 4) == 0.0    assert ndcg_at_k(RANKED, {}, 4) == 0.0

Run it.

.venv/Scripts/python.exe -m pytest test_measure.py -v

Expected output.

============================= test session starts =============================platform win32 -- Python 3.13.9, pytest-9.1.1, pluggy-1.6.0cachedir: .pytest_cacheplugins: anyio-4.14.2collecting ... collected 5 itemstest_measure.py::test_recall_counts_only_what_was_retrieved PASSED       [ 20%]test_measure.py::test_mrr_uses_the_first_hit PASSED                      [ 40%]test_measure.py::test_mrr_is_zero_when_the_hit_is_below_k PASSED         [ 60%]test_measure.py::test_ndcg_is_one_for_a_perfect_ranking PASSED           [ 80%]test_measure.py::test_empty_relevant_does_not_divide_by_zero PASSED      [100%]============================== 5 passed in 0.06s ==============================

Trimmed from that block: pytest also prints your interpreter path on the platform line and a rootdir line, both pointing into your own bakeoff directory. Your platform, versions and timing will differ too. The five test names and 5 passed are what matter.

What just happened. You have a scoring layer that takes a ranked list of document ids and a judgment dictionary, and returns a float. Every arm from here on produces the first thing and reuses this for the second. If test_mrr_is_zero_when_the_hit_is_below_k had failed, the [:k] slice would be missing and every MRR number in the final table would be inflated.

Step 3: Build a BM25 index with bm25s (arm 1)

Goal. Index all 5,183 documents with bm25s and score the run.

Why this step. BM25 costs a millisecond per query and no model at all. It runs more than an order of magnitude cheaper than the dense arm and four orders cheaper than the reranked pipeline, which is why every neural arm has to justify itself against this number. Build it first so that every later arm is measured as an increment over something real.

Code.

File: lexical.py

# lexical.pyimport bm25sfrom corpus import DOCS, QUERIES, QRELSfrom measure import recall_at_k, mrr_at_k, ndcg_at_k_ids = [d["id"] for d in DOCS]_texts = [d["text"] for d in DOCS]_retriever = bm25s.BM25()_retriever.index(bm25s.tokenize(_texts, show_progress=False))def search(query: str, k: int = 10) -> list[str]:    idx, _ = _retriever.retrieve(bm25s.tokenize(query, show_progress=False), k=k)    return [_ids[i] for i in idx[0]]def run_all(k: int = 10) -> dict[str, list[str]]:    return {q["id"]: search(q["text"], k) for q in QUERIES}if __name__ == "__main__":    runs = run_all(10)    n = len(runs)    print(f"arm          BM25")    print(f"queries      {n}")    print(f"Recall@10    {sum(recall_at_k(r, QRELS[q], 10) for q, r in runs.items()) / n:.4f}")    print(f"MRR@10       {sum(mrr_at_k(r, QRELS[q], 10) for q, r in runs.items()) / n:.4f}")    print(f"NDCG@10      {sum(ndcg_at_k(r, QRELS[q], 10) for q, r in runs.items()) / n:.4f}")

Run it.

.venv/Scripts/python.exe lexical.py

Expected output. bm25s writes tqdm progress bars to stderr; this is stdout:

arm          BM25queries      300Recall@10    0.7739MRR@10       0.6312NDCG@10      0.6617

What just happened. search and run_all now exist. Every later arm copies this signature deliberately, so the fusion step can treat any two arms as interchangeable.

The single most likely defect here is _ids drifting out of order with the indexed texts, which produces plausible-looking scores built on nonsense. Check it directly.

File: check_align.py

# check_align.py"""Guard against _ids drifting out of order with the indexed texts.Misaligned ids produce plausible-looking scores built on nonsense, which is theworst kind of bug here because nothing raises. This counts how many queriesretrieved at least one document their own qrels mark relevant. A number nearzero means the mapping is wrong; a number consistent with Recall@10 means it isnot."""from corpus import QRELS, QUERIESfrom lexical import run_allruns = run_all(10)hits = sum(1 for q in QUERIES if any(d in QRELS[q["id"]] for d in runs[q["id"]]))print(f"{hits}/{len(QUERIES)} queries have at least one qrel hit in top 10")

.venv/Scripts/python.exe check_align.py

Expected output:

239/300 queries have at least one qrel hit in top 10

That 239 is consistent with Recall@10 of 0.7739 given that most queries have exactly one relevant document. Misaligned ids would put this near zero.

One number to hold on to: NDCG@10 of 0.6617. Step 8 comes back to it.

These three figures are deterministic on the pinned versions. Expect yours to match to the third decimal. A gap of up to about 0.001 is score ties being broken in a different order, which is harmless and has a fix in section 5. Anything larger than that means something is actually wrong, and the alignment check above is where to start.

Step 4: Add dense retrieval with sentence-transformers (arm 2)

Goal. Encode every document once, rank by cosine similarity, and score the run.

Why this step. Measuring this arm beside BM25 is the whole design of the harness. At 5,183 documents a vector database earns nothing, so a single normalised numpy matrix and a dot product is both simpler and faster than the alternative.

Code.

File: dense.py

# dense.pyimport numpy as npfrom sentence_transformers import SentenceTransformerfrom corpus import DOCS, QUERIES, QRELSfrom measure import recall_at_k, mrr_at_k, ndcg_at_k_ids = [d["id"] for d in DOCS]_model = SentenceTransformer("sentence-transformers/all-MiniLM-L6-v2")_emb = _model.encode(    [d["text"] for d in DOCS], normalize_embeddings=True, show_progress_bar=False)def search(query: str, k: int = 10) -> list[str]:    q = _model.encode([query], normalize_embeddings=True, show_progress_bar=False)    scores = _emb @ q[0]                      # normalized, so dot == cosine    top = np.argsort(-scores)[:k]    return [_ids[i] for i in top]def run_all(k: int = 10) -> dict[str, list[str]]:    return {q["id"]: search(q["text"], k) for q in QUERIES}if __name__ == "__main__":    runs = run_all(10)    n = len(runs)    print(f"arm          Dense (all-MiniLM-L6-v2)")    print(f"queries      {n}")    print(f"Recall@10    {sum(recall_at_k(r, QRELS[q], 10) for q, r in runs.items()) / n:.4f}")    print(f"MRR@10       {sum(mrr_at_k(r, QRELS[q], 10) for q, r in runs.items()) / n:.4f}")    print(f"NDCG@10      {sum(ndcg_at_k(r, QRELS[q], 10) for q, r in runs.items()) / n:.4f}")

Run it. The document encode runs on CPU with no progress output at all, so it looks like a hang. Across three runs on the same machine it took 1.7, 9.1 and about 12 minutes; SciFact abstracts are long, and this is the most load-sensitive step in the tutorial. Fifteen minutes is not a hang. Leave it.

.venv/Scripts/python.exe dense.py

Expected output.

arm          Dense (all-MiniLM-L6-v2)queries      300Recall@10    0.7833MRR@10       0.6047NDCG@10      0.6451

What just happened. You now have two arms disagreeing. Dense takes Recall@10 (0.7833 against 0.7739) while BM25 takes MRR@10 (0.6312 against 0.6047) and NDCG@10 (0.6617 against 0.6451). Neither dominates.

These three numbers are how you know the harness is not lying to you. MTEB, the public leaderboard that publishes per-dataset scores for embedding models, reports this exact model on SciFact at nDCG@10 0.64508, Recall@10 0.78333, MRR@10 0.60472. Ours reproduce all three to four decimal places. Four decimals of agreement across three metrics is not luck: it validates qrels loading, the title-plus-text concatenation, embedding normalisation, the top-k cutoff and all three metric implementations at once. It says nothing about whether the model is any good, only that we are measuring it correctly.

Do this on your own harness before you trust any comparison it prints. Pick one arm you can tie to a published row, reproduce that row, and only then believe the rows nobody else has published. A bake-off you cannot anchor to an external reference is a script, not a harness.

Step 5: Compare BM25 against dense retrieval query by query

Goal. Print the queries with the largest per-query gap between BM25 and dense.

Why this step. Averages will not show you this. Two arms can average within a point of each other and still be right about completely different queries. It is also where this corpus takes apart the lexical-versus-semantic split.

Code.

File: disagree.py

# disagree.py"""Where the two arms disagree. Aggregates hide this; it is the whole point.The tidy story says BM25 wins on rare exact tokens and dense wins on paraphrases.On this corpus that story does not survive contact with the data: rare technicaltokens land on both sides of the split, and neither arm wins outright - densetakes Recall@10, BM25 takes MRR@10 and NDCG@10, and more than half the queriesscore identically. Read the queries below before you trust any rule of thumb,including that one."""from corpus import QUERIES, QRELSfrom measure import mrr_at_kimport lexical, denseK = 10bm25_runs, dense_runs = lexical.run_all(K), dense.run_all(K)rows = []for q in QUERIES:    qid, rel = q["id"], QRELS[q["id"]]    b, d = mrr_at_k(bm25_runs[qid], rel, K), mrr_at_k(dense_runs[qid], rel, K)    rows.append((b - d, q["text"], b, d))rows.sort(key=lambda r: r[0])print(f"{'delta':>7}  {'bm25':>5}  {'dense':>5}  query")print("--- dense wins most ---")for delta, text, b, d in rows[:5]:    print(f"{delta:>7.3f}  {b:>5.3f}  {d:>5.3f}  {text[:58]}")print("--- bm25 wins most ---")for delta, text, b, d in rows[-5:]:    print(f"{delta:>7.3f}  {b:>5.3f}  {d:>5.3f}  {text[:58]}")ties = sum(1 for r in rows if r[0] == 0)print(f"\nqueries where both arms scored identically: {ties} of {len(rows)}")

Run it. Read this before you start it. disagree.py imports lexical and dense directly, so it encodes all 5,183 documents again from scratch - the same silent 2-to-12 minute wait as step 4, with no output until it finishes. Nothing is broken; do not interrupt it.

It will re-encode every time you run it, and doc_emb.npy existing changes nothing. The embedding cache introduced in step 6 lives in stage_runs.py, which installs it around dense at import time. disagree.py imports dense straight, so the cache never reaches it.

Pay a single encode now, or skip this step and read the output block below. disagree.py writes no artifact and nothing downstream imports it.

.venv/Scripts/python.exe disagree.py

Expected output. One command, two blocks below - the script prints a blank line before the tie count, so it renders as two.

  delta   bm25  dense  query--- dense wins most --- -1.000  0.000  1.000  Activator-inhibitor pairs are provided dorsally by Admpcho -1.000  0.000  1.000  Bone marrow cells contribute to adult macrophage compartme -1.000  0.000  1.000  High dietary calcium intakes are unnecessary for preventio -1.000  0.000  1.000  Silencing of Bcl2 is important for the maintenance and pro -1.000  0.000  1.000  The myocardial lineage develops from cardiac progenitors o--- bm25 wins most ---  1.000  1.000  0.000  Crossover hot spots are not found within gene promoters in  1.000  1.000  0.000  Ivermectin is used to treat onchocerciasis.  1.000  1.000  0.000  Macrolides protect against myocardial infarction.  1.000  1.000  0.000  Recurrent mutations occur frequently within CTCF anchor si  1.000  1.000  0.000  Venules have a thinner or absent smooth layer compared to

queries where both arms scored identically: 167 of 300

What just happened. Read the query text, not the deltas.

The tidy story says rare exact tokens go to BM25 and paraphrases go to dense. Rare technical tokens are on both sides here. Admpchordin and Bcl2 are on the dense-wins side; Ivermectin, onchocerciasis, Macrolides, CTCF and Saccharomyces cerevisiae are on the BM25-wins side. Three of those are cut off by the 58-character slice the script prints, so do not go hunting for them in the block above: Saccharomyces cerevisiae ends the crossover-hot-spots query, Admpchordin is truncated to Admpcho, and the calcium query's above 75 nmol/liter threshold is past the cut. Widen the slice in disagree.py if you want to read them in full. Admpchordin is about as rare a token as exists in this set, and BM25 scored zero on it.

Something weaker than a rule is visible in these ten rows. The BM25 wins read like paper titles - "Ivermectin is used to treat onchocerciasis" is six words naming a drug and a disease. The dense wins do not read that way at all: "provided dorsally by", "important for the maintenance and progression of", and a 19-word claim that hangs on a negation and the numeric threshold above 75 nmol/liter. Relational phrasing, conditional phrasing, and clauses whose meaning survives paraphrase. Encoders are supposed to be good at exactly that, and term matching cannot see it at all. That is a tendency across ten queries out of 300. It is not a law, and it still does not explain Admpchordin.

Then there is the number nobody reports: 167 of 300 queries score identically on both arms. On more than half this corpus the choice of retriever changed nothing at all. When people report a lexical-versus-semantic split from an aggregate table, this is the layer they did not look at. Part 3 of the series covers why domain jargon breaks encoders, which is the mechanism the rule of thumb is reaching for - it is just not what splits this corpus in two.

Step 6: Combine two rankings with reciprocal rank fusion (arm 3)

Goal. Combine the BM25 and dense rankings by rank position, and stage all three runs to disk.

Why this step. Section 3 covered why score-based fusion smuggles in a weight you then have to retune. Here is the arithmetic that avoids it: each document's fused score is the sum across arms of 1 / (k_rrf + rank), so nothing needs normalising.

We left k_rrf at 60, the value in Cormack et al.'s original paper and the documented default in Elasticsearch, where the parameter is called rank_constant. We did not vary it. Bruch, Gai and Ingber dispute the folklore that RRF is insensitive to that constant, so treat "RRF needs no tuning" as the literature's claim. This run does not demonstrate it.

This step also splits the harness into stages. Encoding 5,183 documents and cross-encoding 15,000 query-document pairs will not finish in one sitting, so stage 1 writes its runs to runs.json and caches the document embeddings to doc_emb.npy.

Code.

File: fuse.py

# fuse.py"""Reciprocal rank fusion.Worked example: a doc ranked #1 by BM25 and #3 by dense outscores one ranked#1 by BM25 alone but missing from the dense run entirely. Positions are allthis reads, so there is no score scale to reconcile.k_rrf stays at 60 here and is never varied, so this run measures nothingabout how sensitive the fused ranking is to it."""from collections import defaultdictdef rrf(runs: list[dict[str, list[str]]], k_rrf: int = 60) -> dict[str, list[str]]:    fused: dict[str, list[str]] = {}    for qid in runs[0]:        scores: dict[str, float] = defaultdict(float)        for run in runs:            for rank, doc_id in enumerate(run[qid], start=1):                scores[doc_id] += 1.0 / (k_rrf + rank)        fused[qid] = sorted(scores, key=scores.get, reverse=True)    return fused

stage_runs.py runs long. The fingerprinted embedding cache at the top and the three timed calls at the bottom are what matter; skim the rest. It retrieves 50 candidates per arm where earlier steps took 10, because the reranker needs a candidate window to work in and fusion needs depth to find agreement.

File: stage_runs.py

# stage_runs.py"""Stage 1 of the bake-off: BM25, dense, and RRF-fused runs.Does the first three arms - lexical, dense, and their RRF fusion - and writesruns.json for stage_rerank.py (step 7) and bakeoff.py (step 9) to consume. Itdoes no cross-encoding.Caches the encoded document matrix to doc_emb.npy so re-runs skip the encode.The cache is installed by monkey-patching SentenceTransformer.encode *before*importing dense.py, so dense.py itself is untouched - it has no idea a cacheexists. The patch only intercepts the one call that passes all 5183 documenttexts at once; per-query encode calls (a list of length 1) fall through to thereal model untouched.The cache is validated against a fingerprint (doc_emb.meta.json) coveringdocument count, a hash of every document's id+text, and the model name.Matching only on filename plus document count was tried first and rejected:swap the corpus for a same-size one, or swap dense.py's model, and a staledoc_emb.npy would be silently reused - every dense, hybrid, and reranknumber downstream would be wrong with no error. On a fingerprint mismatchthe cache is invalidated and the documents are re-encoded, with a printedmessage explaining why."""import hashlibimport jsonimport osimport timeimport numpy as npfrom corpus import DOCSCANDIDATES = 50DOC_EMB_CACHE = "doc_emb.npy"DOC_EMB_META = "doc_emb.meta.json"RUNS_PATH = "runs.json"# Kept in sync by hand with dense.py's hardcoded SentenceTransformer name -# dense.py is not modified, so this string can't be imported from it without# triggering the encode dense.py runs at import time. If dense.py's model# ever changes, this constant must change with it or the fingerprint will# (correctly) stop protecting against that swap.DENSE_MODEL_NAME = "sentence-transformers/all-MiniLM-L6-v2"def _doc_fingerprint() -> dict:    """Fingerprint of the exact, ORDERED sequence of (id, text) pairs.    Order matters, not just content: doc_emb.npy's row i corresponds to    DOCS[i] (dense.py aligns them via the same list comprehension), so a    reordering of DOCS with identical content would silently misalign the    cached embeddings even though a set-based hash would call it unchanged.    """    h = hashlib.sha256()    for d in DOCS:        h.update(d["id"].encode("utf-8"))        h.update(b"\x01")        h.update(d["text"].encode("utf-8"))        h.update(b"\x00")    return {"count": len(DOCS), "doc_hash": h.hexdigest(), "model_name": DENSE_MODEL_NAME}_expected_fp = _doc_fingerprint()_cache_hit = Falseif os.path.exists(DOC_EMB_CACHE) and os.path.exists(DOC_EMB_META):    with open(DOC_EMB_META) as f:        _stored_fp = json.load(f)    if _stored_fp == _expected_fp:        _cache_hit = True    else:        print(            f"[cache] {DOC_EMB_META} fingerprint mismatch "            f"(stored count={_stored_fp.get('count')} model={_stored_fp.get('model_name')!r}, "            f"current count={_expected_fp['count']} model={_expected_fp['model_name']!r}) "            f"- invalidating {DOC_EMB_CACHE} and re-encoding"        )elif os.path.exists(DOC_EMB_CACHE):    print(f"[cache] {DOC_EMB_CACHE} has no fingerprint sidecar - treating as stale, re-encoding")if _cache_hit:    from sentence_transformers import SentenceTransformer    _cached_emb = np.load(DOC_EMB_CACHE)    _orig_encode = SentenceTransformer.encode    def _cached_encode(self, sentences, **kwargs):        if isinstance(sentences, list) and len(sentences) == len(DOCS):            return _cached_emb        return _orig_encode(self, sentences, **kwargs)    SentenceTransformer.encode = _cached_encode    print(f"[cache] loaded {DOC_EMB_CACHE} - fingerprint matches, skipping document encode")import dense  # noqa: E402  (must import after the optional monkey-patch above)import fuse  # noqa: E402import lexical  # noqa: E402if not _cache_hit:    np.save(DOC_EMB_CACHE, dense._emb)    with open(DOC_EMB_META, "w") as f:        json.dump(_expected_fp, f)    print(f"[cache] saved {DOC_EMB_CACHE} and {DOC_EMB_META}")def timed(fn):    start = time.perf_counter()    result = fn()    return result, time.perf_counter() - startbm25_runs, t_bm25 = timed(lambda: lexical.run_all(CANDIDATES))print(f"BM25  run_all({CANDIDATES}) took {t_bm25:.1f}s")dense_runs, t_dense = timed(lambda: dense.run_all(CANDIDATES))print(f"Dense run_all({CANDIDATES}) took {t_dense:.1f}s")hybrid_runs, t_fuse = timed(lambda: fuse.rrf([bm25_runs, dense_runs]))print(f"Fuse  rrf(...) took {t_fuse:.2f}s")with open(RUNS_PATH, "w") as f:    json.dump(        {            "bm25": bm25_runs,            "dense": dense_runs,            "hybrid": hybrid_runs,            "timings": {"bm25": t_bm25, "dense": t_dense, "fuse": t_fuse},        },        f,    )print(f"[write] {RUNS_PATH} ({len(bm25_runs)} queries)")

Run it.

.venv/Scripts/python.exe stage_runs.py

Expected output. Read this before you compare, because your first run and every run after it differ on line 1.

Your first run pays the step 4 document encode all over again - 1.7 to 12 minutes of silence - and then prints [cache] saved doc_emb.npy and doc_emb.meta.json as its opening line. Every run after that loads the cache instead and starts almost instantly. The block below is a second run:

[cache] loaded doc_emb.npy - fingerprint matches, skipping document encodeBM25  run_all(50) took 0.2sDense run_all(50) took 15.8sFuse  rrf(...) took 0.04s[write] runs.json (300 queries)

So on run one, expect saved where that says loaded, and expect to wait for it. Your three timings will differ either way; ours varied by more than a factor of two between runs on the same machine depending on load, and the timings above come from a different run than the table in step 9. The retrieval content does not vary. We confirmed that by diffing two runs' runs.json: the bm25, dense and hybrid rankings were identical, and only the timings moved.

What just happened. runs.json now holds three complete runs at 50 candidates per query plus the wall-clock time each took, and doc_emb.npy holds the encoded corpus behind a SHA-256 fingerprint of the ordered document ids and text plus the model name. The fingerprint exists because a filename-and-count check is not enough: swap in a different corpus of the same size, or a different 384-dimensional encoder, and stale embeddings would be reused silently. Every dense, hybrid and reranked number after that would be wrong with no error anywhere.

Step 7: Rerank the fused list with a cross-encoder, checkpointed for resume (arm 4)

Goal. Re-score the top 50 fused candidates per query with cross-encoder/ms-marco-MiniLM-L-6-v2 and checkpoint the result.

Why this step. Section 3 covered what a cross-encoder does; this tutorial does not re-teach it. How to Rerank Retrieval Results with a Cross-Encoder builds one from scratch and measures it, and Part 4 of the series covers the theory. Here it is one contestant in a table, and the only one whose cost you will feel while waiting.

Code. Two files. rerank.py re-scores one candidate window and is deliberately thin. stage_rerank.py is the checkpointing wrapper around it: it walks the 300 queries in batches of 25, appends to rerank.json after every batch, and skips any query already present. The candidate window is top_n=50, matching what stage_runs.py retrieved.

File: rerank.py

# rerank.py"""Arm 4. Deliberately thin - the cross-encoder tutorial explains why this works.Here it is only a contestant in the table."""from sentence_transformers import CrossEncoderfrom corpus import DOCS, QUERIES_text = {d["id"]: d["text"] for d in DOCS}_query = {q["id"]: q["text"] for q in QUERIES}_model = CrossEncoder("cross-encoder/ms-marco-MiniLM-L-6-v2")def rerank(runs: dict[str, list[str]], top_n: int = 50) -> dict[str, list[str]]:    out = {}    for qid, ranked in runs.items():        cand = ranked[:top_n]        scores = _model.predict([(_query[qid], _text[c]) for c in cand])        out[qid] = [c for _, c in sorted(zip(scores, cand), reverse=True)]    return out

File: stage_rerank.py

# stage_rerank.py"""Stage 2 of the bake-off: cross-encode the hybrid run, checkpointed.Cross-encoding ~300 queries x 50 candidates on CPU does not fit inside oneforeground command's time budget - SciFact abstracts are long, so each(query, doc) pair is not cheap. This script processes queries in batches of25, writing the accumulated results to rerank.json after EVERY batch, so arun that gets cut off mid-way loses at most one batch of work.Safe and correct to run repeatedly: on startup it loads whateverrerank.json already has, skips queries already done, reports how manyremain, and picks up where it left off.Measured batch times across three full passes on this box ranged 158-356s (25queries x 50 candidates each), varying with machine load - not a tight band,and wide enough that batch count is the only thing worth predicting.TIME_BUDGET_S bounds how many batches are allowed to START, checked once atthe top of each loop iteration; it does NOT bound wall-clock, since a batchthat has already started runs to completion regardless of the budget. Twobatches under this budget can still add up to more than TIME_BUDGET_S ofwall time. Pick TIME_BUDGET_S with that slop in mind, and rely on thecheckpoint - not the budget - as the actual safety net: whatever doesn'tfinish this invocation just runs again next time.rerank.json is also fingerprinted against the hybrid run it was computedfrom (a hash of every qid's ranked candidate list, stored as "hybrid_hash").Without this, regenerating runs.json (e.g. after a corpus or model change)would leave every qid "already done" against a hybrid run rerank.json wasnever actually computed from - bakeoff.py would score stale reranked listsagainst new hybrid runs and report "complete" with no error. On a mismatch,rerank.json is invalidated and reranking starts over from zero, with aprinted message explaining why."""import hashlibimport jsonimport osimport timefrom rerank import rerankRUNS_PATH = "runs.json"OUT_PATH = "rerank.json"BATCH_SIZE = 25TIME_BUDGET_S = 300with open(RUNS_PATH) as f:    staged = json.load(f)hybrid_runs = staged["hybrid"]def _hybrid_fingerprint(runs: dict[str, list[str]]) -> str:    h = hashlib.sha256()    for qid in sorted(runs):        h.update(qid.encode("utf-8"))        h.update(b"\x01")        h.update("|".join(runs[qid]).encode("utf-8"))        h.update(b"\x00")    return h.hexdigest()hybrid_hash = _hybrid_fingerprint(hybrid_runs)if os.path.exists(OUT_PATH):    with open(OUT_PATH) as f:        state = json.load(f)    if state.get("hybrid_hash") != hybrid_hash:        print(            f"[cache] {OUT_PATH}'s hybrid_hash does not match the current "            f"runs.json hybrid run - the candidates being reranked have "            f"changed since rerank.json was written. Invalidating and "            f"starting reranking over from zero."        )        state = {"reranked": {}, "cross_encoder_seconds": 0.0, "hybrid_hash": hybrid_hash}else:    state = {"reranked": {}, "cross_encoder_seconds": 0.0, "hybrid_hash": hybrid_hash}remaining_qids = [qid for qid in hybrid_runs if qid not in state["reranked"]]print(f"{len(remaining_qids)} queries remaining")start = time.perf_counter()processed = 0for i in range(0, len(remaining_qids), BATCH_SIZE):    if time.perf_counter() - start > TIME_BUDGET_S:        print(f"[budget] stopping after {processed} queries this invocation")        break    batch_qids = remaining_qids[i : i + BATCH_SIZE]    batch = {qid: hybrid_runs[qid] for qid in batch_qids}    batch_start = time.perf_counter()    batch_out = rerank(batch, top_n=50)    batch_elapsed = time.perf_counter() - batch_start    state["reranked"].update(batch_out)    state["cross_encoder_seconds"] += batch_elapsed    with open(OUT_PATH, "w") as f:        json.dump(state, f)    processed += len(batch_qids)    print(        f"[batch] {len(batch_qids)} queries in {batch_elapsed:.1f}s "        f"(done: {len(state['reranked'])}/{len(hybrid_runs)}, "        f"cumulative cross-encoder time: {state['cross_encoder_seconds']:.1f}s)"    )still_remaining = len(hybrid_runs) - len(state["reranked"])if still_remaining == 0:    print("0 queries remaining - rerank.json is complete")else:    print(f"{still_remaining} queries remaining - run stage_rerank.py again")

Run it. Repeatedly, until it reports zero remaining. If you are on macOS or Linux and picking this up in a fresh terminal, remember the prefix is .venv/bin/python. The first full pass took 2,699.8 seconds of cross-encoder time here, spread over twelve batches of 25 queries. A second full pass on the same machine took 2,608.1 seconds over the same twelve batches, which is the run the output below comes from. Expect six or seven invocations, or up to twelve if your batches are slow enough that only one fits per run: TIME_BUDGET_S lets two batches start per invocation, and two batches of roughly 215 seconds each overrun a 300-second budget together. The first invocation downloads the cross-encoder before it prints anything at all, and each batch then takes three to six minutes in silence. So the first [batch] line is a while coming, and each invocation after that runs seven to eleven minutes.

.venv/Scripts/python.exe stage_rerank.py

Expected output. Each invocation prints how many queries remain when it starts, one [batch] line per completed batch with a running cross-encoder total, and a closing status line. Watch the done: counter. The invocation that finishes the last batch looks like this:

50 queries remaining[batch] 25 queries in 217.5s (done: 275/300, cumulative cross-encoder time: 2390.0s)[batch] 25 queries in 218.1s (done: 300/300, cumulative cross-encoder time: 2608.1s)0 queries remaining - rerank.json is complete

0 queries remaining - rerank.json is complete is the line to stop on. Run the script once more and it confirms there is nothing left to do:

0 queries remaining0 queries remaining - rerank.json is complete

Your batch times will differ, and so may the shape of that block. TIME_BUDGET_S bounds how many batches are allowed to start, so on a faster pass two batches fit per invocation and on a slower one only a single batch does. We have seen the completing invocation open at 50 queries remaining with two [batch] lines and at 25 queries remaining with one. Both are correct. Two things are invariant: twelve batches, because 300 queries in batches of 25 is always twelve, and the closing line you stop on.

What just happened. rerank.json holds a reranked 50-document list for all 300 queries, fingerprinted against the hybrid run it came from. The first full pass here took 45 minutes. A 45-minute pass that cannot resume is one you kill at minute 40 and never come back to.

Step 8: Enable Snowball stemming in bm25s (arm 5)

Goal. Rebuild the BM25 index with a Snowball stemmer and score it as a separate arm.

Why this step. Our 0.6617 lands on the bm25s paper's no-stemmer row (0.662), not the stemmed row (0.687). Stemming there is opt-in in the tokenizer, not in the retriever, so the default configuration is unstemmed and nothing warns you. That gap is 0.025 NDCG@10, and every neural arm compared against the lower row inherits it as free credit.

One naming quirk before the code: you installed PyStemmer but you import Stemmer. That is the package's own convention.

The trap to avoid: index time and query time must use the same stemmer. If they do not, the query vocabulary will not line up with the index vocabulary and retrieval returns garbage without raising anything.

Code.

File: lexical_stemmed.py

# lexical_stemmed.py"""BM25 with Snowball English stemming enabled - sibling of lexical.py.Stemming is opt-in in bm25s's tokenizer, not in the retriever itself: pass astemmer to bm25s.tokenize() and both the index build and every query mustuse the *same* stemmer, or the query-side vocabulary won't line up with theindex-side vocabulary and every retrieval silently returns garbage."""import bm25simport Stemmerfrom corpus import DOCS, QUERIES, QRELSfrom measure import recall_at_k, mrr_at_k, ndcg_at_k_ids = [d["id"] for d in DOCS]_texts = [d["text"] for d in DOCS]_stemmer = Stemmer.Stemmer("english")_retriever = bm25s.BM25()_retriever.index(    bm25s.tokenize(_texts, stemmer=_stemmer, show_progress=False),    create_empty_token=True,)def search(query: str, k: int = 10) -> list[str]:    # Same stemmer as index time, and create_empty_token=True at index time    # means a query that stems to zero vocabulary tokens (short queries can,    # once stopwords and stemming both strip them down) still retrieves    # instead of raising "The query does not contain any tokens that are in    # the vocabulary."    idx, _ = _retriever.retrieve(        bm25s.tokenize(query, stemmer=_stemmer, show_progress=False), k=k    )    return [_ids[i] for i in idx[0]]def run_all(k: int = 10) -> dict[str, list[str]]:    return {q["id"]: search(q["text"], k) for q in QUERIES}if __name__ == "__main__":    runs = run_all(10)    n = len(runs)    print(f"arm          BM25 + stemmer")    print(f"queries      {n}")    print(f"Recall@10    {sum(recall_at_k(r, QRELS[q], 10) for q, r in runs.items()) / n:.4f}")    print(f"MRR@10       {sum(mrr_at_k(r, QRELS[q], 10) for q, r in runs.items()) / n:.4f}")    print(f"NDCG@10      {sum(ndcg_at_k(r, QRELS[q], 10) for q, r in runs.items()) / n:.4f}")

Run it.

.venv/Scripts/python.exe lexical_stemmed.py

Expected output.

arm          BM25 + stemmerqueries      300Recall@10    0.8187MRR@10       0.6492NDCG@10      0.6863

What just happened. One flag moved NDCG@10 from 0.6617 to 0.6863 and Recall@10 from 0.7739 to 0.8187. The 0.6863 is 0.0007 below the bm25s paper's stemmed SciFact row, which is inside run-to-run noise and confirms the tokenizer is configured the way we think it is.

This arm now beats the dense arm on all three metrics, and lands within a hundredth of NDCG@10 of both neural rows. Read that as a warning about baselines. It is not a claim that stemming beats neural retrieval in general. An unstemmed BM25 baseline understates the lexical arm on this corpus by roughly 0.025 NDCG@10, and every comparison built on it inherits that error.

One caveat you must carry into the next step: the hybrid and reranked arms in the final table are built on the unstemmed BM25 run from runs.json. We did not fuse or rerank on top of the stemmed arm. That experiment is still open.

Step 9: Score all five arms and print the comparison table

Goal. Score all five arms and print one comparison table.

Why this step. Every arm so far printed its own three numbers in isolation. The comparison is the artifact. bakeoff.py does no model work for four of the five arms: it reads runs.json and rerank.json and scores them. The stemmed arm is timed live, because BM25 retrieval is fast enough that staging it would be pointless.

Code. Read CANDIDATES and HOURLY_USD before you run it:

• CANDIDATES = 50. Each single arm retrieved 50 candidates. The fused hybrid list is the union of two such lists, so it runs longer - between 63 and 99 entries on this corpus - and the reranker re-scored its top 50. The table then scores the top 10 from every arm.
• HOURLY_USD = 0.05. The $/1k q column is HOURLY_USD / 3600 * seconds_per_query * 1000. It is a linear rescale of the latency column by one constant you chose. Change it to your own rate. It is not a cost claim, and it deliberately excludes the one-time document encode.

File: bakeoff.py

# bakeoff.py - the finished harness. Five arms, one table.## Reads what stage_runs.py (runs.json) and stage_rerank.py (rerank.json)# already staged and does no model work of its own for those four arms - it# only scores them. The split exists because encoding 5183 docs and# cross-encoding ~15,000 (query, doc) pairs cannot both finish inside one# foreground command's time budget; see stage_runs.py and stage_rerank.py# for where the real work happens.## The exception is the "BM25 + stemmer" row: BM25 indexing and retrieval is# ~0.4ms/query, so building that index and timing it here - rather than# staging it - still finishes instantly. It is a genuinely separate arm,# measured on its own, purely to show what a Snowball stemmer buys a lexical# retriever. The hybrid (RRF) and rerank rows below are NOT rebuilt on top of# it - they still fuse and rerank the original, unstemmed BM25 run from# runs.json. If you want a hybrid/rerank pipeline built on stemmed BM25,# that's a separate experiment, not this row.import jsonimport timeimport lexical_stemmedfrom corpus import QRELSfrom measure import mrr_at_k, ndcg_at_k, recall_at_kK = 10CANDIDATES = 50            # each arm retrieved/reranked this many candidates                            # (see stage_runs.py, stage_rerank.py) before this                            # script scores the top K from eachHOURLY_USD = 0.05          # a small always-on CPU box; change to your own ratewith open("runs.json") as f:    staged = json.load(f)with open("rerank.json") as f:    rerank_state = json.load(f)bm25_runs = staged["bm25"]dense_runs = staged["dense"]hybrid_runs = staged["hybrid"]reranked = rerank_state["reranked"]t_bm25 = staged["timings"]["bm25"]t_dense = staged["timings"]["dense"]t_fuse = staged["timings"]["fuse"]t_rank = rerank_state["cross_encoder_seconds"]missing = set(hybrid_runs) - set(reranked)if missing:    raise SystemExit(        f"rerank.json is incomplete - {len(missing)} queries not yet reranked. "        f"Run stage_rerank.py again before bakeoff.py."    )# The reranked list is what stage_rerank.py's top_n=CANDIDATES truncation# actually produced - verify it, rather than just asserting CANDIDATES in# a comment nobody checks.assert all(len(r) == CANDIDATES for r in reranked.values()), (    f"expected every reranked list to have exactly {CANDIDATES} candidates")# Measured here, not staged: lexical_stemmed's module-level index build# (~1s for 5183 docs) already ran on import above; this only times retrieval,# same as lexical.run_all(CANDIDATES) is timed in stage_runs.py._t0 = time.perf_counter()stemmed_runs = lexical_stemmed.run_all(CANDIDATES)t_stemmed = time.perf_counter() - _t0def evaluate(name: str, runs: dict[str, list[str]], elapsed: float) -> None:    n = len(runs)    per_query_ms = elapsed / n * 1000    cost_per_1k = HOURLY_USD / 3600 * (elapsed / n) * 1000    print(        f"{name:<22}"        f"{sum(recall_at_k(r, QRELS[q], K) for q, r in runs.items()) / n:>10.4f}"        f"{sum(mrr_at_k(r, QRELS[q], K) for q, r in runs.items()) / n:>10.4f}"        f"{sum(ndcg_at_k(r, QRELS[q], K) for q, r in runs.items()) / n:>10.4f}"        f"{per_query_ms:>12.1f}{cost_per_1k:>12.5f}"    )print(f"# {CANDIDATES} candidates retrieved per arm, reranked, and scored at K={K}\n")print(f"{'arm':<22}{'Recall@10':>10}{'MRR@10':>10}{'NDCG@10':>10}{'ms/query':>12}{'$/1k q':>12}")evaluate("BM25", bm25_runs, t_bm25)evaluate("BM25 + stemmer", stemmed_runs, t_stemmed)evaluate("Dense", dense_runs, t_dense)evaluate("Hybrid (RRF)", hybrid_runs, t_bm25 + t_dense + t_fuse)evaluate("Hybrid + rerank", reranked, t_bm25 + t_dense + t_fuse + t_rank)

Run it.

.venv/Scripts/python.exe bakeoff.py

Expected output.

# 50 candidates retrieved per arm, reranked, and scored at K=10arm                    Recall@10    MRR@10   NDCG@10    ms/query      $/1k qBM25                      0.7739    0.6312    0.6617         0.4     0.00001BM25 + stemmer            0.8187    0.6492    0.6863         0.6     0.00001Dense                     0.7833    0.6047    0.6451        24.3     0.00034Hybrid (RRF)              0.8352    0.6571    0.6962        24.8     0.00034Hybrid + rerank           0.8239    0.6608    0.6931      9024.2     0.12534

What just happened. Hybrid RRF takes two of the three quality columns - Recall@10 and NDCG@10 - at essentially dense's latency, because fusion adds a dictionary walk to work the dense arm had already done. That much is what the harness was built to produce, and it is unsurprising. MRR@10 goes to the reranked row instead, by 0.0037, and step 10 is about where that came from.

Two rows are not.

The stemmer row costs nothing measurable over plain BM25. On one of our runs it came out 0.2 ms per query faster than the unstemmed arm. It lands 0.0068 NDCG@10 under the bottom row, which runs four orders of magnitude slower per query, and 0.0099 under the best row. If you are choosing where to spend an engineering week, those ratios are the finding.

The bottom row is also worse than the row above it on two of three quality metrics, which step 10 takes apart.

We quote orders of magnitude, not exact multiples, on purpose. Both lexical rows sit under two milliseconds per query, close enough to the timer's noise floor that a precise ratio against them is not a measurement. We ran the table on a second machine: our own quality columns reproduced exactly, to four decimals, while the stemmer-to-reranker ratio moved between roughly 6,000 and 15,000 depending on load. The orders of magnitude held. The exact multiples did not, so we do not quote them.

A note on the latency column before you over-read it. These are wall-clock means on one busy laptop CPU. Do not cite them. Our own timings varied by roughly half across repeated runs of the same code. The quality columns are deterministic and reproduce to the third decimal on these pins; the latency and cost columns are indicative. Treat the four orders of magnitude between BM25 and the cross-encoder as real, and the difference between 24.3 and 24.8 as noise.

Step 10: Diagnose why reranking lowered Recall@10

Goal. Split the hybrid-to-reranked delta by how many relevant documents each query has.

Why this step. The bottom row gained MRR@10 and lost Recall@10 and NDCG@10. Those three metrics usually move together, so when they split, one of them is measuring something the others cannot see. Averages over 300 queries will not tell you which. Splitting the query set is what turns that confusing row into something you can act on.

Code.

File: decompose.py

# decompose.py"""Split the hybrid -> rerank delta by how many relevant docs a query has.The bake-off table says reranking raised MRR@10 and lowered Recall@10 andNDCG@10. A mean over 300 queries cannot tell you which queries moved. Thissplits the set on the one property that changes what MRR can even see:whether a query has exactly one judged-relevant document, or more than one.Reads runs.json and rerank.json only. No model work, no re-retrieval."""import jsonfrom corpus import QRELSfrom measure import mrr_at_k, ndcg_at_k, recall_at_kK = 10with open("runs.json") as f:    hybrid = json.load(f)["hybrid"]with open("rerank.json") as f:    reranked = json.load(f)["reranked"]single = [q for q in hybrid if len(QRELS[q]) == 1]multi = [q for q in hybrid if len(QRELS[q]) >= 2]def delta(bucket, metric):    return sum(        metric(reranked[q], QRELS[q], K) - metric(hybrid[q], QRELS[q], K)        for q in bucket    ) / len(bucket)print(f"{'bucket':<18}{'n':>5}{'Recall@10':>12}{'MRR@10':>12}{'NDCG@10':>12}")for name, bucket in (("single-relevant", single), ("multi-relevant", multi)):    print(        f"{name:<18}{len(bucket):>5}"        f"{delta(bucket, recall_at_k):>12.4f}"        f"{delta(bucket, mrr_at_k):>12.4f}"        f"{delta(bucket, ndcg_at_k):>12.4f}"    )lost = gained = 0for q in single:    had = recall_at_k(hybrid[q], QRELS[q], K) > 0    has = recall_at_k(reranked[q], QRELS[q], K) > 0    if had and not has:        lost += 1    elif not had and has:        gained += 1print(f"\nsingle-relevant queries that LOST their only relevant doc:   {lost}")print(f"single-relevant queries that GAINED their only relevant doc: {gained}")print(f"net change in queries with their relevant doc in the top 10: {gained - lost}")

Run it.

.venv/Scripts/python.exe decompose.py

Expected output.

bucket                n   Recall@10      MRR@10     NDCG@10single-relevant     277     -0.0108     -0.0006     -0.0034multi-relevant       23     -0.0174      0.0551      0.0013single-relevant queries that LOST their only relevant doc:   14single-relevant queries that GAINED their only relevant doc: 11net change in queries with their relevant doc in the top 10: -3

What just happened. On the 277 queries with exactly one relevant document - 92% of the set - reranking made all three metrics worse. The entire MRR@10 gain in the final table comes from the 23 multi-relevant queries, where MRR credits only the first relevant document found and is structurally blind to the ones reranking pushed out of the top 10. The reranker's only measurable win came from a metric that stops looking after the first hit, on 7.7% of the query set.

Those three "worse" numbers are one number. On a single-relevant, binary-judgment query, with r the rank of the one relevant document, reciprocal rank is 1/r, NDCG@10 is 1/log2(r+1) for r at most 10, and Recall@10 is 1 if r is at most 10. All three are strictly decreasing functions of the same scalar. So "all three got worse on the single-relevant queries" is one fact reported three ways, not three findings. That is exactly what makes the contrast with the multi-relevant bucket sharp: there, the metrics genuinely decouple.

The churn is larger than the net suggests. Fourteen single-relevant queries lost their only relevant document from the top 10 and eleven different ones gained theirs. The reranker is reshuffling roughly one in ten single-relevant queries in both directions, and the net of -3 is the small residue of a large movement. "Only three queries regressed" would be a much tidier and much less true sentence.

Why it happened, as narrowly as we can support it. The RRF ordering being reranked already scores NDCG@10 0.6962. BEIR's published BM25-plus-cross-encoder result on SciFact is 0.688, achieved with ms-marco-electra-base, a cross-encoder of about 110M parameters. Ours is ms-marco-MiniLM-L-6-v2 at 22.7M. We asked a 22.7M-parameter reranker to improve on an ordering that already beat what a 110M-parameter reranker achieves on this corpus. There was about -0.008 of room, which is to say none.

Now the boundaries on that claim. Reranking does work on SciFact: BEIR reports it lifting the corpus from 0.665 to 0.688, with a larger model. The sort direction is correct; we checked it against a live scoring probe before believing the row. And the failure mode already has a literature. Jacob et al. documented reranking degrading recall in Drowning in Documents, at ReNeuIR at SIGIR 2025, and classified 53.3% of their academic experiments as "helps and scaling hurts". SciFact is in their set.

Their caveats are load-bearing, though. They tested jina, bge, cohere, voyage and gpt-4o-mini rerankers over single-retriever first stages; ours is a MiniLM over a fusion. And their degradation shows up at very large candidate windows where ours is a fixed 50. So they corroborate the mechanism and the metric that suffers without covering this setup. We found nothing at all reporting ms-marco-MiniLM-L-6-v2 degrading on SciFact specifically, which is why this measurement is here as data of its own.

The transferable rule is short. Before you buy a reranker, measure how much room it has. If your first-stage ordering already exceeds the published reranked baseline for your corpus, a small reranker has nothing to add and a top-10 window to damage.

5. Troubleshooting: the errors this harness actually raises

You probably arrived at one of the next two by pasting it into a search box.

The query does not contain any tokens that are in the vocabulary. A query stemmed down to nothing. Pass create_empty_token=True at index time. It is already the default in bm25s 0.3.10, so you only meet this error if you turned it off - lexical_stemmed.py passes it explicitly to document the intent.

k of {k} is larger than the number of available scores, which is {num_docs} You asked for more candidates than you have documents. Clamp with k = min(k, len(DOCS)).

The rest, with the verbatim string in each case:

6. Architecture: how the harness files connect

Section 3 described four ways of deciding what relevant means. This is what you actually built.

flowchart TD
    C["corpus.py<br/>5183 docs / 300 queries / 339 qrels"]
    L["lexical.py<br/>BM25, unstemmed"]
    LS["lexical_stemmed.py<br/>BM25 + Snowball"]
    D["dense.py<br/>all-MiniLM-L6-v2"]
    F["fuse.py<br/>RRF, k=60"]
    R["rerank.py<br/>cross-encoder, top 50"]
    S1["stage_runs.py<br/>writes runs.json"]
    S2["stage_rerank.py<br/>writes rerank.json"]
    M["measure.py<br/>Recall / MRR / NDCG"]
    B["bakeoff.py<br/>five-arm table"]
    DC["decompose.py<br/>single vs multi-relevant"]
    DG["disagree.py<br/>per-query BM25 vs dense"]

    C --> L
    C --> LS
    C --> D
    C --> R
    C --> S1
    C --> DG
    C --> DC
    C --> B
    M --> L
    M --> LS
    M --> D
    M --> DG
    M --> DC
    M --> B
    L --> S1
    D --> S1
    F --> S1
    L --> DG
    D --> DG
    S1 --> S2
    R --> S2
    S1 --> B
    S2 --> B
    LS --> B
    S1 --> DC
    S2 --> DC

    style C fill:#4A90E2,color:#FFFFFF
    style L fill:#98D8C8,color:#2C2C2A
    style LS fill:#6BCF7F,color:#2C2C2A
    style D fill:#98D8C8,color:#2C2C2A
    style F fill:#7B68EE,color:#FFFFFF
    style R fill:#E74C3C,color:#FFFFFF
    style S1 fill:#FFD93D,color:#2C2C2A
    style S2 fill:#FFD93D,color:#2C2C2A
    style M fill:#95A5A6,color:#FFFFFF
    style B fill:#C2185B,color:#FFFFFF
    style DC fill:#FFA07A,color:#2C2C2A
    style DG fill:#FFA07A,color:#2C2C2A

The yellow staging nodes exist because the expensive work has to survive being interrupted. stage_runs.py caches the encoded corpus behind a content fingerprint; stage_rerank.py checkpoints after every batch of 25 queries and fingerprints its output against the hybrid run it came from. The pink node loads no neural model at all: it scores the staged JSON and builds only the cheap stemmed BM25 index, so a full re-score takes a few seconds while you argue about which metric matters.

The two orange diagnostic scripts are not equally cheap, and the arrows say why. decompose.py reads runs.json and rerank.json, so it is instant. disagree.py imports lexical and dense directly, which means it re-runs both arms live and re-encodes the corpus - budget the same time you budgeted for step 4.

Notice too that the green stemmed arm feeds only bakeoff.py. It is not upstream of fusion or reranking. Those two rows are built on the unstemmed BM25 run.

7. Full harness code and file tree

bakeoff/├── requirements.txt     # prerequisites├── check_env.py         # prerequisites├── corpus.py            # step 1  - DOCS, QUERIES, QRELS├── check_qrels.py       # step 1  - judgment-count spread├── check_align.py       # step 3  - id-alignment guard├── measure.py           # step 2  - recall_at_k, mrr_at_k, ndcg_at_k├── test_measure.py      # step 2├── lexical.py           # step 3  - arm 1├── dense.py             # step 4  - arm 2├── disagree.py          # step 5  - per-query BM25 vs dense├── fuse.py              # step 6  - arm 3├── stage_runs.py        # step 6  -> runs.json, doc_emb.npy, doc_emb.meta.json├── rerank.py            # step 7  - arm 4├── stage_rerank.py      # step 7  -> rerank.json├── lexical_stemmed.py   # step 8  - arm 5├── bakeoff.py           # step 9  - the table└── decompose.py         # step 10 - hybrid vs reranked, by bucket

Every file was written in full at the step that needed it, so nothing here is new. This is the resync copy: if your table does not match, diff bakeoff.py against this listing first, since it is the one file that reads every artifact the other fourteen produce.

File: bakeoff.py

# bakeoff.py - the finished harness. Five arms, one table.## Reads what stage_runs.py (runs.json) and stage_rerank.py (rerank.json)# already staged and does no model work of its own for those four arms - it# only scores them. The split exists because encoding 5183 docs and# cross-encoding ~15,000 (query, doc) pairs cannot both finish inside one# foreground command's time budget; see stage_runs.py and stage_rerank.py# for where the real work happens.## The exception is the "BM25 + stemmer" row: BM25 indexing and retrieval is# ~0.4ms/query, so building that index and timing it here - rather than# staging it - still finishes instantly. It is a genuinely separate arm,# measured on its own, purely to show what a Snowball stemmer buys a lexical# retriever. The hybrid (RRF) and rerank rows below are NOT rebuilt on top of# it - they still fuse and rerank the original, unstemmed BM25 run from# runs.json. If you want a hybrid/rerank pipeline built on stemmed BM25,# that's a separate experiment, not this row.import jsonimport timeimport lexical_stemmedfrom corpus import QRELSfrom measure import mrr_at_k, ndcg_at_k, recall_at_kK = 10CANDIDATES = 50            # each arm retrieved/reranked this many candidates                            # (see stage_runs.py, stage_rerank.py) before this                            # script scores the top K from eachHOURLY_USD = 0.05          # a small always-on CPU box; change to your own ratewith open("runs.json") as f:    staged = json.load(f)with open("rerank.json") as f:    rerank_state = json.load(f)bm25_runs = staged["bm25"]dense_runs = staged["dense"]hybrid_runs = staged["hybrid"]reranked = rerank_state["reranked"]t_bm25 = staged["timings"]["bm25"]t_dense = staged["timings"]["dense"]t_fuse = staged["timings"]["fuse"]t_rank = rerank_state["cross_encoder_seconds"]missing = set(hybrid_runs) - set(reranked)if missing:    raise SystemExit(        f"rerank.json is incomplete - {len(missing)} queries not yet reranked. "        f"Run stage_rerank.py again before bakeoff.py."    )# The reranked list is what stage_rerank.py's top_n=CANDIDATES truncation# actually produced - verify it, rather than just asserting CANDIDATES in# a comment nobody checks.assert all(len(r) == CANDIDATES for r in reranked.values()), (    f"expected every reranked list to have exactly {CANDIDATES} candidates")# Measured here, not staged: lexical_stemmed's module-level index build# (~1s for 5183 docs) already ran on import above; this only times retrieval,# same as lexical.run_all(CANDIDATES) is timed in stage_runs.py._t0 = time.perf_counter()stemmed_runs = lexical_stemmed.run_all(CANDIDATES)t_stemmed = time.perf_counter() - _t0def evaluate(name: str, runs: dict[str, list[str]], elapsed: float) -> None:    n = len(runs)    per_query_ms = elapsed / n * 1000    cost_per_1k = HOURLY_USD / 3600 * (elapsed / n) * 1000    print(        f"{name:<22}"        f"{sum(recall_at_k(r, QRELS[q], K) for q, r in runs.items()) / n:>10.4f}"        f"{sum(mrr_at_k(r, QRELS[q], K) for q, r in runs.items()) / n:>10.4f}"        f"{sum(ndcg_at_k(r, QRELS[q], K) for q, r in runs.items()) / n:>10.4f}"        f"{per_query_ms:>12.1f}{cost_per_1k:>12.5f}"    )print(f"# {CANDIDATES} candidates retrieved per arm, reranked, and scored at K={K}\n")print(f"{'arm':<22}{'Recall@10':>10}{'MRR@10':>10}{'NDCG@10':>10}{'ms/query':>12}{'$/1k q':>12}")evaluate("BM25", bm25_runs, t_bm25)evaluate("BM25 + stemmer", stemmed_runs, t_stemmed)evaluate("Dense", dense_runs, t_dense)evaluate("Hybrid (RRF)", hybrid_runs, t_bm25 + t_dense + t_fuse)evaluate("Hybrid + rerank", reranked, t_bm25 + t_dense + t_fuse + t_rank)

Every number in this tutorial came from that script on a laptop CPU. No GPU was used at any point, including the cross-encoder pass. The ms/query column is the argument: four of the five arms finish in tens of milliseconds per query on commodity CPU, and the fifth is three to four orders of magnitude slower because it runs 50 forward passes per query, not because it needs different hardware. Rent a GPU when you have measured that you need one.

8. Where to go next: your own corpus, and two open experiments

Swap in your own corpus

The harness is useful when the corpus is yours. corpus.py is the only file you replace: produce DOCS, QUERIES and QRELS in the same shapes and everything downstream works unchanged.

The labels are the hard part. Fifty labelled queries were enough to separate the arms on our own corpus, and cost a day of two people's time. We never tested how far below fifty you can go. Here is how to build them:

1. Take the queries from your logs. Sample across the whole frequency distribution, tail included, and keep the ones that returned nothing. Those are where a retriever change shows up first.
2. Judge documents, not answers. For each query, mark every document that would let a competent human answer it. You are grading retrieval here; whether the model then writes a good answer is a separate experiment needing a separate harness.
3. Pool the candidates. Run BM25 and dense at k=50 and judge the union - judging the whole corpus is impossible, and judging one arm's output hands that arm the win.
4. Write down what "relevant" means before you label anything, then have a second person label ten queries against your definition. If you disagree on two or three of the ten, the definition is the problem, not the labelling, and it will not resolve itself over the remaining forty. Decide on paper whether a document that supports the claim only in its abstract counts or the body must carry it, whether a document that contradicts the claim counts as relevant to it, and whether you are grading on a scale or a yes/no. SciFact takes the strict path on all three - 339 judgments, every one of them grade 1, a median of one relevant document per query - which is exactly why its Recall@10 behaves almost like a hit rate. Whichever way you go, decide once and write it at the top of the labelling sheet.
5. Check the distribution by running check_qrels.py from step 1 against your own qrels.

Then re-run the harness in this order, which is the only sequence that satisfies the dependencies:

.venv/Scripts/python.exe stage_runs.py      # re-encodes, rewrites runs.json.venv/Scripts/python.exe stage_rerank.py    # repeat until it reports zero remaining.venv/Scripts/python.exe bakeoff.py.venv/Scripts/python.exe decompose.py

Both fingerprints work in your favour here. stage_runs.py sees a new corpus hash and re-encodes. stage_rerank.py sees a new hybrid_hash and restarts reranking from zero. You do not have to remember to delete anything.

The harness assumes at least 50 documents, because CANDIDATES = 50 and bakeoff.py asserts every reranked list is exactly that long. On a smaller collection, lower CANDIDATES in stage_runs.py, the top_n=50 argument in stage_rerank.py's rerank(...) call, and CANDIDATES in bakeoff.py together. If you swap the encoder in dense.py, change DENSE_MODEL_NAME in stage_runs.py to match, or the embedding fingerprint stops protecting you against the one mistake it exists to catch.

Build DOCS in a deterministic order that is identical on every process start. The fingerprint hashes the ordered id-and-text sequence, so a loader that iterates a set produces a fresh hash each run and misses the cache every time. I lost most of an afternoon to that one: the encode ran on every single invocation and nothing anywhere said why. Keep the unjudged-query filter on the last line of corpus.py too. Every arm indexes QRELS[q] directly, and a query with no judgments scores zero on all five while dragging every mean down equally.

Document ids and query ids must be strings, not integers or UUID objects - the staged runs round-trip through JSON, so ids become object keys and every later QRELS[q] lookup happens on the string that came back. decompose.py needs both of its buckets populated: it splits on whether a query has exactly one judged-relevant document or more than one, and divides by each bucket size. If you follow the labelling advice above and land on a median of one, you may have no multi-relevant queries at all - in which case skip that command and read only the single-relevant row.

Two follow-up experiments: fuse the stemmed BM25 arm, vary k_rrf

We measured five arms and left two obvious ones on the floor. Fuse the stemmed arm instead of the unstemmed one: hybrid RRF at 0.6962 was built on a lexical arm we now know was 0.0246 NDCG@10 below its own tokenizer's ceiling. And vary k_rrf: we left it at 60 and never touched it, so this harness has no evidence about how sensitive fusion is to it on this corpus. Both are a few lines in stage_runs.py.

What this harness cannot tell you

It measures retrieval over a fixed document set. That leaves out most of what breaks RAG in production.

It cannot see chunking. Every document here is a whole abstract, so the harness never faces the question of how to split a 40-page PDF, and I have watched chunking decide more retrieval outcomes than any retriever swap did. This harness cannot measure that, so take it as experience rather than a number off the table. Part 2 of the series covers that failure directly. Staleness is invisible too: qrels are frozen and your index is not. And it cannot see context assembly: retrieving the right ten documents and then handing the model a badly ordered prompt is a different failure with the same symptom.

When you have chosen an arm, Building a Full-Stack Hybrid Search System builds the chosen thing for real, with Docker and a serving layer.

9. References

• Thakur, N., Reimers, N., Rücklé, A., Srivastava, A., & Gurevych, I. (2021). BEIR: A Heterogeneous Benchmark for Zero-shot Evaluation of Information Retrieval Models. arXiv:2104.08663 - SciFact corpus statistics and the BM25 and BM25+CE baselines cited here.
• Lù, X. H. (2024). BM25S: Orders of magnitude faster lexical search via eager sparse scoring. arXiv:2407.03618 - the stemmed and unstemmed SciFact ablation rows in step 8.
• Jacob, M., Lindgren, E., Zaharia, M., Carbin, M., Khattab, O., & Drozdov, A. (2025). Drowning in Documents: Consequences of Scaling Reranker Inference. ReNeuIR at SIGIR 2025. arXiv:2411.11767
• Rosa, G., et al. (2022). In Defense of Cross-Encoders for Zero-Shot Retrieval. arXiv:2212.06121
• Bruch, S., Gai, S., & Ingber, A. (2023). An Analysis of Fusion Functions for Hybrid Retrieval. ACM TOIS. arXiv:2210.11934 - the finding that RRF is sensitive to its parameters.
• Cormack, G. V., Clarke, C. L. A., & Büttcher, S. (2009). Reciprocal Rank Fusion outperforms Condorcet and individual Rank Learning Methods. SIGIR 2009, pp. 758-759. DOI 10.1145/1571941.1572114
• MTEB results for sentence-transformers/all-MiniLM-L6-v2 on SciFact, model revision 8b3219a, MTEB v1.12.75: embeddings-benchmark/results
• Official docs: bm25s 0.3.10, sentence-transformers 5.7.0, ir_datasets 0.6.3, PyStemmer 3.1.0
• Model cards: all-MiniLM-L6-v2, ms-marco-MiniLM-L6-v2 (22.7M parameters). The code loads cross-encoder/ms-marco-MiniLM-L-6-v2, with the hyphen before the 6; the repo was renamed and both strings resolve to this same card.

Related Articles

AI Engineering

• Epistemic Restraint by Design: Boundary-Aware AI Systems

Information Retrieval

• How to Rerank Retrieval Results with a Cross-Encoder
• Building Hybrid Search That Actually Works: BM25 + Dense Retrieval + Cross-Encoders
• BM25 for Developers: What Actually Matters in Production