How to Rerank Retrieval Results with a Cross-Encoder

What you'll build

Add a cross-encoder reranker to your retrieval pipeline and measure the real NDCG@10 gain yourself, on a corpus you build, not a vendor benchmark. A bi-encoder retrieves 20 candidates, a cross-encoder reranks them, and you measure the difference before and after.

You will finish with two numbers you generated yourself: how much ranking quality the reranker buys, and what it costs in latency. Intermediate level, about 30 minutes, CPU only.

Here is the finished pipeline's output:

code

corpus passages     : 36queries             : 8candidates per query: 20NDCG@10 retrieve only : 0.8447NDCG@10 after rerank  : 0.9228NDCG@10 delta         : +0.0782

Verified against Python 3.13.9, sentence-transformers 5.3.0, transformers 5.3.0, torch 2.11.0+cpu, numpy 2.3.4, scikit-learn 1.8.0, on 2026-08-06.

Prerequisites

You should know Python and have used text embeddings or a vector store. You do not need to know how cross-encoders work internally, or how NDCG is computed - both are covered here.

Everything runs on CPU. No GPU, no API key, no paid service. The two models are about 90 MB each, so roughly 180 MB downloads on first run.

code

pip install "sentence-transformers==5.3.0" "torch==2.11.0" "numpy==2.3.4"

scikit-learn arrives automatically as a dependency of sentence-transformers, which is where NDCG comes from. You do not install it separately.

Verify the install before starting:

code

python -c "import sentence_transformers, sklearn, torch; print(sentence_transformers.__version__, sklearn.__version__, torch.__version__)"

Expected output - your scikit-learn and torch patch versions may differ:

code

5.3.0 1.8.0 2.11.0+cpu

One version note that will save you an hour. The sentence-transformers documentation site documents 5.6.x, but the API changed in 5.4. On 5.6.x, CrossEncoder.predict() takes its first argument as inputs; on 5.3.0 it is sentences. Copying predict(inputs=...) from the current docs raises TypeError on 5.3.0. Every call in this tutorial passes that argument positionally, which is the only form that is correct and silent on both.

Bi-encoder vs. cross-encoder: why you need both stages

A bi-encoder and a cross-encoder answer different questions.

A bi-encoder turns each passage into a vector once, ahead of time, and turns your query into a vector at search time. Relevance is the cosine similarity between the two vectors. Because the passage vectors are precomputed, you can search millions of passages in milliseconds. The cost of that speed is that the query and the passage never meet: each was compressed into a vector without knowledge of the other. You will watch that happen in step 2. The bi-encoder puts a passage about thread mutexes above the one about transaction lock ordering, and its top four scores land inside 0.043 of each other. It is not confidently wrong. It cannot see a difference.

A cross-encoder takes a query and a passage together, as one input, and runs both through a transformer that lets every query token attend to every passage token. It returns a single relevance score. Nothing is precomputed, so you run the model once per query-passage pair. Far too slow to search a corpus; far more accurate on a small set.

Two-stage retrieval uses each where it is strong: the bi-encoder narrows the corpus to a shortlist, the cross-encoder reorders it. The reranker never sees the passages the retriever missed. That ceiling is permanent, and it is why the last thing you tune in this tutorial is the shortlist size and not the model.

The measurement is NDCG@10 - normalized discounted cumulative gain over the top 10 results. It rewards relevant items near the top and discounts them the further down they sit, normalized against the best possible ordering, so 1.0 is a perfect ranking. You will compute it once on the bi-encoder's ordering and again on the reranked ordering.

Step 1: Build a corpus with planted decoys

Make an empty directory first and run every command in this tutorial from inside it. All eight files sit side by side, and the imports assume that.

Goal. Create a small corpus and a set of labeled queries.

Why this step. A reranker has nothing to fix unless the first stage is already wrong. A corpus of unrelated passages will not show anything, because the bi-encoder will already rank it perfectly. You need passages that share vocabulary with a query while answering a different question. Those are the ones the bi-encoder ranks too highly, and they are what the cross-encoder is for.

Code. Create corpus.py. This is the whole file - copy it as it stands. It is long because the corpus is the data the rest of the tutorial measures, and a shortened version would not reproduce any of the numbers below.

code

# corpus.py"""A small, hand-built corpus about database and distributed-systems operations.Deliberately planted with lexical decoys: passages that share vocabulary with aquery but answer a different question. Those are the cases a bi-encoder rankstoo highly and a cross-encoder pushes back down."""CORPUS = [    # --- replication lag: 1 answer, several decoys sharing vocabulary ---    "A read replica falls behind when the primary produces write-ahead log records "    "faster than the replica can apply them. The usual cause is single-threaded "    "replay on the replica against a parallel write workload on the primary. Enable "    "parallel apply workers, or reduce long transactions on the primary that force "    "the replica to serialize.",    "The primary key index is rebuilt automatically after a bulk load completes. "    "Rebuilding blocks writes to the table for the duration, so schedule bulk loads "    "outside peak hours.",    "Consumer lag in a message queue is the gap between the newest offset in a "    "partition and the offset a consumer group has committed. Lag grows when "    "consumers process slower than producers publish.",    "Choosing a primary key with a random distribution spreads writes evenly across "    "shards. A monotonically increasing primary key concentrates every insert on "    "one shard and creates a write hotspot.",    "Replica promotion during failover discards any write-ahead log records the "    "replica had not yet applied. Measure replication lag before promoting, or "    "those writes are lost.",    # --- deadlocks: 1 answer, decoys sharing "lock" and "deadlock" ---    "Two transactions deadlock when each holds a lock the other needs. The database "    "detects the cycle and aborts one transaction with a deadlock error. Fix it by "    "making every transaction acquire locks on tables in the same order, so no "    "cycle can form.",    "Lock contention is not a deadlock. Contention means transactions wait for each "    "other and eventually proceed; a deadlock means they wait forever and one must "    "be aborted. High contention shows up as increased latency, not as errors.",    "A thread deadlock in application code happens when two threads each hold a "    "mutex the other wants. This is unrelated to database transaction deadlocks and "    "is not visible in database logs.",    "Row-level locks are held until the transaction commits or rolls back, not "    "until the statement finishes. A transaction that reads a row early and commits "    "much later holds that lock for its whole lifetime.",    "Advisory locks are acquired explicitly by the application and are not tied to "    "any row or table. The database will not detect a deadlock cycle that involves "    "only advisory locks.",    # --- connection pool exhaustion ---    "Connection pool exhaustion shows up as clients blocking on checkout while the "    "database itself sits idle. The pool has no free connections because existing "    "ones are held open by slow queries or by transactions that were never "    "committed. Cap transaction duration and set a checkout timeout so the failure "    "is loud.",    "Increasing the pool size past what the database can serve makes throughput "    "worse, not better. Each additional connection costs memory and scheduler time "    "on the server, and the queue simply moves from the client to the database.",    "A connection leak means the application borrowed a connection and never "    "returned it. Pool metrics show active connections climbing and never falling, "    "even when traffic drops.",    # --- index selection ---    "A composite index on (tenant_id, created_at) serves queries that filter on "    "tenant_id alone and queries that filter on both columns. It does not serve a "    "query filtering only on created_at, because the leading column is missing.",    "An index on a low-cardinality column, such as a boolean status flag, is rarely "    "used. The planner estimates that scanning the table costs less than reading "    "the index and then fetching most of the rows anyway.",    "Adding an index speeds up reads and slows down writes. Every insert, update, "    "and delete must maintain every index on the table, so an unused index is pure "    "write overhead.",    "A partial index covers only the rows matching a predicate. For a table where "    "99 percent of rows are archived, an index restricted to the active rows is a "    "fraction of the size and stays in memory.",    "Index-only scans avoid touching the table when every column the query needs is "    "present in the index. Adding a frequently selected column to the index payload "    "can turn a slow query into a fast one without changing the query.",    # --- query planner / statistics ---    "The query planner chooses a plan from estimated row counts, not actual ones. "    "When statistics are stale, the estimate can be off by orders of magnitude and "    "the planner picks a nested loop where a hash join was correct.",    "Running ANALYZE refreshes the statistics the planner uses. After a large data "    "load, statistics describe the old table and plans degrade until ANALYZE runs.",    "A sequential scan is not automatically a problem. On a small table, or when a "    "query returns most rows, a sequential scan is the cheapest plan and forcing an "    "index scan makes it slower.",    # --- partitioning / sharding ---    "Range partitioning by time lets old partitions be dropped in constant time "    "instead of deleted row by row. A DELETE of a billion rows generates a billion "    "write-ahead log records; dropping a partition generates almost none.",    "Cross-shard joins require pulling data from multiple shards to one "    "coordinator. The coordinator becomes the bottleneck and the join cost grows "    "with the number of shards, not with the result size.",    "Resharding requires moving data while it is being written. Doing this without "    "downtime needs dual writes to both the old and new placement, plus a backfill "    "and a verification pass before the cutover.",    # --- caching ---    "A cache stampede happens when a popular key expires and every concurrent "    "request recomputes it at once. Use a short lock around the recompute, or "    "refresh the value before it expires, so only one request does the work.",    "Cache invalidation on write is only correct if the write and the invalidation "    "cannot be reordered. If the invalidation lands before the write commits, the "    "cache immediately refills with the old value.",    "A cache hit rate of 99 percent can still leave the database overloaded. What "    "matters is the absolute number of misses per second, not the ratio.",    # --- backups / durability ---    "A backup that has never been restored is not a backup. Restore drills catch "    "missing extensions, permission mismatches, and version skew that a successful "    "backup job does not reveal.",    "Point-in-time recovery needs both a base backup and the continuous write-ahead "    "log archive. Losing the archive means recovery can only reach the moment the "    "base backup was taken.",    "Synchronous replication trades write latency for durability. Every commit "    "waits for at least one replica to acknowledge, so a slow replica directly "    "slows down the primary.",    # --- transactions / isolation ---    "Read committed isolation allows non-repeatable reads: the same row read twice "    "in one transaction can return different values. Code that reads a value, "    "decides on it, and writes it back needs a stronger isolation level or explicit "    "locking.",    "Serializable isolation does not make transactions run one at a time. It runs "    "them concurrently and aborts the ones whose interleaving could not have "    "happened serially, so the application must be prepared to retry.",    "A long-running read transaction prevents vacuum from reclaiming dead rows, "    "because those rows may still be visible to it. Table bloat during a long "    "analytics query is the usual symptom.",    # --- observability ---    "Average query latency hides the problem. A p50 of 5 milliseconds with a p99 of "    "4 seconds means one request in a hundred is timing out, and the average will "    "not show it.",    "Query fingerprinting groups statements that differ only in their literal "    "values. Without it, a workload of parameterized queries appears as millions of "    "unique statements and no pattern is visible.",    "Slow query logs capture statements that exceeded a threshold, which means they "    "systematically miss the fast query executed ten thousand times per second. "    "Total time, not per-call time, identifies that one.",]# Graded relevance. Keys are zero-based indices into CORPUS; values are the grade:# 2 = directly answers the query, 1 = related and useful, 0 = everything else (omitted).## The queries are deliberately terse, the way people actually type into a search# box. Short queries are where bi-encoders struggle most: with few tokens to work# with, surface vocabulary overlap dominates the embedding, and a passage that# merely repeats the query's words outranks the one that answers it.QUERIES = [    {        # Decoys: passage 1 ("primary key index"), 3 ("primary key"), 2 ("lag").        "query": "primary lag",        "relevant": {0: 2, 4: 1},    },    {        # Decoys: 7 (thread deadlock), 9 (advisory locks, no cycle detection).        "query": "deadlock lock order",        "relevant": {5: 2, 6: 1},    },    {        # Decoy: 11 ("increasing the pool size"), 12 ("connection leak").        "query": "pool exhausted database idle",        "relevant": {10: 2},    },    {        # Decoys: 13, 15, 16, 17 all discuss indexes.        "query": "index boolean column not used",        "relevant": {14: 2},    },    {        # Decoy: 20 ("sequential scan is not automatically a problem").        "query": "nested loop wrong after load",        "relevant": {18: 2, 19: 2},    },    {        # Decoy: 22 (cross-shard joins), 23 (resharding) both mention partitions.        "query": "drop old rows write ahead log volume",        "relevant": {21: 2},    },    {        # Decoys: 25 (cache invalidation), 26 (cache hit rate).        "query": "cache key expired concurrent recompute",        "relevant": {24: 2},    },    {        # Decoy: 34 (fingerprinting), 35 (slow query logs) are both observability.        "query": "average latency fine users time out",        "relevant": {33: 2},    },]

The queries are deliberately terse, the way people actually type into a search box. Short queries are where bi-encoders struggle most: with few tokens to work with, surface vocabulary overlap dominates the embedding.

Run it.

code

python -c "from corpus import CORPUS, QUERIES; print(len(CORPUS), 'passages,', len(QUERIES), 'queries')"

Expected output.

code

36 passages, 8 queries

What just happened. You have a corpus and a graded relevance judgement for each query. Those grades are the ground truth every NDCG number in this tutorial is measured against, so read one entry before moving on. Query 2 is deadlock lock order with {5: 2, 6: 1}: passage 5 answers it directly, passage 6 explains the contrast with lock contention and is useful but not the answer, and the thread-deadlock passage gets nothing.

Step 2: Retrieve candidates with a bi-encoder

Goal. Rank the corpus for one query using embeddings alone, and look at what comes back.

Why this step. This is the baseline. Every improvement you claim later is measured against this ordering, so you need to see it before you change it. You could use BM25 here instead, and on keyword-heavy corpora you probably should. Dense retrieval wins on this corpus because the decoys share vocabulary with the queries, which is exactly where BM25 is blind. The fuller comparison is here.

Code. Create retrieve.py.

code

# retrieve.pyfrom sentence_transformers import SentenceTransformerfrom sentence_transformers.util import semantic_searchfrom corpus import CORPUSQUERY = "deadlock lock order"retriever = SentenceTransformer("sentence-transformers/all-MiniLM-L6-v2")corpus_emb = retriever.encode(CORPUS, convert_to_tensor=True, show_progress_bar=False)query_emb = retriever.encode([QUERY], convert_to_tensor=True, show_progress_bar=False)hits = semantic_search(query_emb, corpus_emb, top_k=5)[0]for rank, hit in enumerate(hits, 1):    print(f"{rank}. [{hit['score']:.4f}] {CORPUS[hit['corpus_id']][:60]}...")

semantic_search takes a batch of query embeddings and returns one result list per query, which is what the trailing [0] selects - there is only one query here. Each hit is a dict with corpus_id, the index into CORPUS, and score, the cosine similarity. That same [0] appears in every later script for the same reason.

Run it. The first run downloads the model, so expect a pause of a minute or two.

If the progress bar sits at 0 bytes and never moves, or you see an OSError mentioning huggingface.co, that is not the pause. It is one of two known download failures. Both have one-line fixes in Fixing the errors you'll hit running this further down. Jump there rather than waiting.

code

python retrieve.py

Expected output.

code

1. [0.5452] A thread deadlock in application code happens when two threa...2. [0.5307] Two transactions deadlock when each holds a lock the other n...3. [0.5186] Lock contention is not a deadlock. Contention means transact...4. [0.5022] Advisory locks are acquired explicitly by the application an...5. [0.4477] Row-level locks are held until the transaction commits or ro...

What just happened. The bi-encoder put the wrong passage first. Rank 1 is about thread deadlocks in application code - a passage that says outright it is unrelated to database deadlocks. The passage that actually answers the query sits at rank 2. The four top scores are packed between 0.5452 and 0.5022, which is the shape of a model that cannot tell these passages apart.

Step 3: Measure the baseline with NDCG@10

Goal. Turn that ordering into a single number.

Why this step. "The wrong passage is first" does not scale to eight queries, and it gives you nothing to compare against later. NDCG@10 collapses a ranking into one score between 0 and 1, so you can measure a change instead of eyeballing it. NDCG@10 measures the ranking, not the answer. A pipeline can rank perfectly and still generate something wrong, which is a separate measurement problem.

Code. Create measure.py.

code

# measure.pyimport numpy as npfrom sklearn.metrics import ndcg_scoredef ndcg_at_k(rows_true, rows_score, k=10):    """NDCG@k over all queries at once.    scikit-learn's ndcg_score expects a 2-D array: one row per query. A flat    1-D list raises ValueError. One query as a single row is fine, which is    what step 3 does; batching all eight is just cheaper.    """    return float(ndcg_score(np.array(rows_true), np.array(rows_score), k=k))

Code. Create baseline.py. It scores the query from step 2 over the top 20 candidates rather than the top 5.

20 is the shortlist stage 2 will rerank: deep enough that the answer is almost always inside it for a 36-passage corpus, shallow enough that the cross-encoder runs 20 times instead of 36. It is the knob you tune at the end.

code

# baseline.pyfrom sentence_transformers import SentenceTransformerfrom sentence_transformers.util import semantic_searchfrom corpus import CORPUS, QUERIESfrom measure import ndcg_at_kTOP_K = 20retriever = SentenceTransformer("sentence-transformers/all-MiniLM-L6-v2")corpus_emb = retriever.encode(CORPUS, convert_to_tensor=True, show_progress_bar=False)item = QUERIES[1]                      # "deadlock lock order"query, grades = item["query"], item["relevant"]query_emb = retriever.encode([query], convert_to_tensor=True, show_progress_bar=False)hits = semantic_search(query_emb, corpus_emb, top_k=TOP_K)[0]cand_ids = [hit["corpus_id"] for hit in hits]true_row = [grades.get(cid, 0) for cid in cand_ids]score_row = [float(hit["score"]) for hit in hits]print(f"query      : {query!r}")print(f"grades     : {true_row[:6]} ...")print(f"NDCG@10    : {ndcg_at_k([true_row], [score_row]):.4f}")

Run it.

code

python baseline.py

Expected output.

code

query      : 'deadlock lock order'grades     : [0, 2, 1, 0, 0, 0] ...NDCG@10    : 0.6697

What just happened. The baseline for this query is 0.6697, not 1.0, because the relevant passage sits at rank 2 instead of rank 1 and NDCG discounts it for that. You now have a number to beat.

Read the grades line before you move on. [0, 2, 1, ...] is the relevance of each candidate in the order stage 1 returned them. Position 0 is the thread-deadlock decoy at grade 0, position 1 is the real answer at grade 2, position 2 is the lock-contention passage at grade 1. That row is also your integrity check on step 1. If your grades line differs, your corpus paste is misaligned and every number below will drift.

Where 0.6697 comes from

You were promised NDCG would be explained, not just called, so here is the whole calculation on the row you just printed.

Ranks here are 1-based, so the grade at position 0 of that row sits at rank 1. Each position contributes its grade divided by log2(rank + 1), which for position 0 is log2(2). Summing the top 10:

code

DCG  = 0/log2(2) + 2/log2(3) + 1/log2(4) + 0 ...     = 0 + 1.2619 + 0.5     = 1.7619

The ideal ordering puts the grade-2 passage first and the grade-1 passage second:

code

IDCG = 2/log2(2) + 1/log2(3)     = 2.0 + 0.6309     = 2.6309

code

NDCG@10 = 1.7619 / 2.6309 = 0.6697

One detail that trips people coming from the literature: scikit-learn uses the grade itself as the gain. Many papers use 2^grade - 1, which on this same row gives 0.6590 instead. Neither is wrong, but they are not comparable, so do not put a scikit-learn number next to a published one and call it a match.

A second detail matters more, and step 5 depends on it. ndcg_score does not assume the row you hand it is already in ranked order. It sorts the grades by the matching score row, then applies the discount. That is why step 5 can pass the same grade rows twice and get two different numbers - only the scores change, and the scores are what induce the ordering. Here the two coincided, because stage 1 returned its candidates already sorted by score.

Note the shape the helper forces: relevance rows and score rows are both lists of lists. Passing one query as a flat list raises ValueError: Only ('multilabel-indicator', 'continuous-multioutput', 'multiclass-multioutput') formats are supported. Got multiclass instead.

Step 4: Rerank the candidates with a cross-encoder

Goal. Rescore the same 20 candidates with a cross-encoder and look at the new order.

Why this step. The bi-encoder scored the query and the passage separately. The cross-encoder reads them together, which is the only way to notice that a passage about thread mutexes does not answer a question about transaction lock ordering.

Code. Create rerank.py.

code

# rerank.pyimport numpy as npfrom sentence_transformers import CrossEncoder, SentenceTransformerfrom sentence_transformers.util import semantic_searchfrom corpus import CORPUSQUERY = "deadlock lock order"retriever = SentenceTransformer("sentence-transformers/all-MiniLM-L6-v2")reranker = CrossEncoder("cross-encoder/ms-marco-MiniLM-L6-v2")corpus_emb = retriever.encode(CORPUS, convert_to_tensor=True, show_progress_bar=False)query_emb = retriever.encode([QUERY], convert_to_tensor=True, show_progress_bar=False)hits = semantic_search(query_emb, corpus_emb, top_k=20)[0]cand_ids = [hit["corpus_id"] for hit in hits]# Pass the pairs positionally. The keyword is `sentences` on 5.3.x and# `inputs` on 5.4+, so positional is the only form correct on both.scores = reranker.predict([(QUERY, CORPUS[cid]) for cid in cand_ids])for rank, i in enumerate(np.argsort(scores)[::-1][:5], 1):    print(f"{rank}. [{scores[i]:+.4f}] {CORPUS[cand_ids[i]][:60]}...")

Run it.

code

python rerank.py

Expected output.

code

1. [+2.6957] Two transactions deadlock when each holds a lock the other n...2. [+2.5766] Lock contention is not a deadlock. Contention means transact...3. [+0.4505] Advisory locks are acquired explicitly by the application an...4. [+0.0930] A thread deadlock in application code happens when two threa...5. [-3.8228] Row-level locks are held until the transaction commits or ro...

What just happened. The correct passage moved from rank 2 to rank 1, and the thread-deadlock decoy fell from rank 1 to rank 4. Look at the score spread: the bi-encoder's top four sat inside 0.043 of each other, while the cross-encoder separates the same passages by more than 2.6 points. Cross-encoder scores are raw logits, not probabilities, so they are unbounded and can be negative. Only their order matters.

Step 5: Measure the reranked pipeline, honestly

Goal. Run both stages over all eight queries and compare NDCG@10 before and after.

Why this step. One query proves nothing. Aggregate measurement is also the only way to discover that reranking does not help uniformly - and on some queries makes things worse.

Code. Create pipeline.py. It loops over every query, retrieves 20 candidates, records the stage-1 ordering, reranks, and scores both orderings with the helper from step 3.

code

# pipeline.pyfrom sentence_transformers import CrossEncoder, SentenceTransformerfrom sentence_transformers.util import semantic_searchfrom corpus import CORPUS, QUERIESfrom measure import ndcg_at_kTOP_K = 20   # how many candidates stage 1 hands to stage 2AT_K = 10    # the k in NDCG@10def main():    retriever = SentenceTransformer("sentence-transformers/all-MiniLM-L6-v2")    reranker = CrossEncoder("cross-encoder/ms-marco-MiniLM-L6-v2")    corpus_emb = retriever.encode(CORPUS, convert_to_tensor=True, show_progress_bar=False)    true_rows, stage1_rows, stage2_rows = [], [], []    for item in QUERIES:        query, grades = item["query"], item["relevant"]        query_emb = retriever.encode([query], convert_to_tensor=True, show_progress_bar=False)        hits = semantic_search(query_emb, corpus_emb, top_k=TOP_K)[0]        cand_ids = [hit["corpus_id"] for hit in hits]        true_rows.append([grades.get(cid, 0) for cid in cand_ids])        stage1_rows.append([float(hit["score"]) for hit in hits])        scores = reranker.predict([(query, CORPUS[cid]) for cid in cand_ids])        stage2_rows.append([float(score) for score in scores])    before = ndcg_at_k(true_rows, stage1_rows, k=AT_K)    after = ndcg_at_k(true_rows, stage2_rows, k=AT_K)    print(f"corpus passages     : {len(CORPUS)}")    print(f"queries             : {len(QUERIES)}")    print(f"candidates per query: {TOP_K}")    print()    print(f"NDCG@{AT_K} retrieve only : {before:.4f}")    print(f"NDCG@{AT_K} after rerank  : {after:.4f}")    print(f"NDCG@{AT_K} delta         : {after - before:+.4f}")if __name__ == "__main__":    main()

Run it.

code

python pipeline.py

Expected output.

code

corpus passages     : 36queries             : 8candidates per query: 20NDCG@10 retrieve only : 0.8447NDCG@10 after rerank  : 0.9228NDCG@10 delta         : +0.0782

What just happened. Reranking moved NDCG@10 from 0.8447 to 0.9228 across the corpus. The delta prints as +0.0782 where subtracting the two displayed figures gives 0.0781 - it is computed on the unrounded values.

The aggregate hides the interesting part. Create per_query.py to see it:

code

# per_query.pyfrom sentence_transformers import CrossEncoder, SentenceTransformerfrom sentence_transformers.util import semantic_searchfrom corpus import CORPUS, QUERIESfrom measure import ndcg_at_kTOP_K = 20retriever = SentenceTransformer("sentence-transformers/all-MiniLM-L6-v2")reranker = CrossEncoder("cross-encoder/ms-marco-MiniLM-L6-v2")corpus_emb = retriever.encode(CORPUS, convert_to_tensor=True, show_progress_bar=False)print(f"{'query':40} {'before':>7} {'after':>7} {'delta':>8}")for item in QUERIES:    query, grades = item["query"], item["relevant"]    query_emb = retriever.encode([query], convert_to_tensor=True, show_progress_bar=False)    hits = semantic_search(query_emb, corpus_emb, top_k=TOP_K)[0]    cand_ids = [hit["corpus_id"] for hit in hits]    true_row = [grades.get(cid, 0) for cid in cand_ids]    before = ndcg_at_k([true_row], [[float(h["score"]) for h in hits]])    scores = reranker.predict([(query, CORPUS[cid]) for cid in cand_ids])    after = ndcg_at_k([true_row], [[float(s) for s in scores]])    print(f"{query:40} {before:7.4f} {after:7.4f} {after - before:+8.4f}")

code

python per_query.py

code

query                                     before   after    deltaprimary lag                               0.4744  0.4628  -0.0116deadlock lock order                       0.6697  1.0000  +0.3303pool exhausted database idle              1.0000  1.0000  +0.0000index boolean column not used             1.0000  1.0000  +0.0000nested loop wrong after load              0.6131  0.9197  +0.3066drop old rows write ahead log volume      1.0000  1.0000  +0.0000cache key expired concurrent recompute    1.0000  1.0000  +0.0000average latency fine users time out       1.0000  1.0000  +0.0000

The gain is not evenly spread:

Query	Before	After	Delta
`deadlock lock order`	0.6697	1.0000	+0.3303
`nested loop wrong after load`	0.6131	0.9197	+0.3066
`primary lag`	0.4744	0.4628	-0.0116
five other queries	1.0000	1.0000	+0.0000

Five queries were already perfect, so reranking had no room to help. Two improved sharply. One got slightly worse: primary lag is ambiguous enough that a passage about message-queue consumer lag is a defensible answer, and the cross-encoder keeps it first. That is not a bug in your code. A reranker can lower quality on queries the first stage already got right, and the only way to know your net position is to measure it the way you just did. That failure is quiet in production, which I have written about in why rerankers fail silently.

What it costs

The opening promised a number on whether the reranker earned its latency. Quality is only half of that. Create timing.py:

code

# timing.pyimport timefrom sentence_transformers import CrossEncoder, SentenceTransformerfrom sentence_transformers.util import semantic_searchfrom corpus import CORPUS, QUERIESTOP_K = 20retriever = SentenceTransformer("sentence-transformers/all-MiniLM-L6-v2")reranker = CrossEncoder("cross-encoder/ms-marco-MiniLM-L6-v2")corpus_emb = retriever.encode(CORPUS, convert_to_tensor=True, show_progress_bar=False)retrieve_s = rerank_s = 0.0pairs = 0for item in QUERIES:    query = item["query"]    t0 = time.perf_counter()    query_emb = retriever.encode([query], convert_to_tensor=True, show_progress_bar=False)    hits = semantic_search(query_emb, corpus_emb, top_k=TOP_K)[0]    retrieve_s += time.perf_counter() - t0    cand_ids = [hit["corpus_id"] for hit in hits]    t0 = time.perf_counter()    reranker.predict([(query, CORPUS[cid]) for cid in cand_ids])    rerank_s += time.perf_counter() - t0    pairs += len(cand_ids)n = len(QUERIES)print(f"retrieve : {retrieve_s / n * 1000:6.1f} ms per query")print(f"rerank   : {rerank_s / n * 1000:6.1f} ms per query ({TOP_K} pairs each)")print(f"ratio    : {rerank_s / retrieve_s:6.1f}x slower")print(f"rerank throughput: {pairs / rerank_s:.0f} pairs/sec")

code

python timing.py

This is the one place in the tutorial where your numbers will not match mine, and they are not a checkpoint. Absolute timings depend on your CPU, your thermal state, and what else is running. On my Windows laptop with the pinned CPU-only build, two consecutive runs gave:

code

retrieve :   19.0 ms per queryrerank   :  365.1 ms per query (20 pairs each)ratio    :   19.2x slowerrerank throughput: 55 pairs/sec

code

retrieve :   22.0 ms per queryrerank   :  415.0 ms per query (20 pairs each)ratio    :   18.9x slowerrerank throughput: 48 pairs/sec

The absolute figures moved by 14 percent between just these two runs, and by more across a longer sample. The ratio did not. Reranking 20 candidates cost roughly 19 times what retrieving them did, on both runs. That ratio is the durable number, and it is the one to carry into a latency budget.

Now the tradeoff is a decision rather than a feeling: +0.0782 NDCG@10 for about 19x the per-query cost, on a shortlist of 20. Whether that is worth it depends on your traffic and your p99 target, and you now have both halves measured.

Fixing the errors you'll hit running this

Fix: TypeError: CrossEncoder.predict() got an unexpected keyword argument 'inputs'

This is the first error most people hit, because it is what you get by copying predict(inputs=...) from the current documentation site onto the pinned 5.3.0 stack. Pass the pairs positionally instead: reranker.predict(pairs). The full list of errors follows.

What you see	Cause	Fix
`TypeError: CrossEncoder.predict() got an unexpected keyword argument 'inputs'`	You copied `predict(inputs=...)` from the current docs, which document 5.6.x. On 5.3.0 the parameter is `sentences`.	Pass the pairs positionally: `reranker.predict(pairs)`.
`ValueError: Only ('multilabel-indicator', 'continuous-multioutput', 'multiclass-multioutput') formats are supported. Got multiclass instead`	You called `ndcg_score` with a single query as a flat list.	Batch queries as rows of one 2-D array, as `ndcg_at_k` does.
The model download hangs at 0 bytes, with the `blobs` folder empty	Xet-backed downloads stall on some Windows setups.	Set the env var before running: `$env:HF_HUB_DISABLE_XET = "1"` in PowerShell, `export HF_HUB_DISABLE_XET=1` in bash.
`OSError: We couldn't connect to 'https://huggingface.co' to load this file, couldn't find it in the cached files`	First run needs network to fetch the models.	Run once online. After that, pass `local_files_only=True` to both constructors.
`Token indices sequence length is longer than the specified maximum sequence length for this model`	A query-passage pair exceeds the cross-encoder's 512-token window.	Informational - the library truncates for you. But the truncated tail is invisible to the reranker, so chunk long passages before indexing.
`NotImplementedError: Cannot copy out of meta tensor; no data!`	Passing `device=` at construction, or constructing the same model twice in one process.	Construct once per process and omit `device=`, then call `.to("cpu")` if you need it.
`RuntimeError: operator torchvision::nms does not exist`	Mismatched `torch` and `torchvision` builds. `transformers` imports `torchvision` eagerly even for text work.	Uninstall `torchvision` if you only use text models, or reinstall matching builds.

The full two-stage pipeline, end to end

You have now built both stages. This is the shape of the finished pipeline. The two NDCG figures on it are the corpus-wide numbers from step 5, not single-query values - for deadlock lock order alone the pair is 0.6697 and 1.0000.

mermaid

flowchart TD
    Q["Query: deadlock lock order"] --> BE["Bi-encoder<br/>all-MiniLM-L6-v2"]
    C["36 passages"] --> EMB["Corpus embeddings<br/>computed once, cached"]
    EMB --> SS["semantic_search<br/>cosine similarity"]
    BE --> SS
    SS --> CAND["Top 20 candidates<br/>NDCG@10 = 0.8447"]
    CAND --> CE["Cross-encoder<br/>ms-marco-MiniLM-L6-v2<br/>20 query-passage pairs"]
    CE --> FINAL["Reranked top 10<br/>NDCG@10 = 0.9228"]

    style Q fill:#4A90E2,color:#FFFFFF
    style C fill:#95A5A6,color:#FFFFFF
    style BE fill:#98D8C8,color:#2C2C2A
    style EMB fill:#98D8C8,color:#2C2C2A
    style SS fill:#7B68EE,color:#FFFFFF
    style CAND fill:#FFD93D,color:#2C2C2A
    style CE fill:#C2185B,color:#FFFFFF
    style FINAL fill:#6BCF7F,color:#2C2C2A

The asymmetry between the two stages is the whole design. The corpus embeddings on the left are computed once and reused for every query forever. The cross-encoder on the right runs 20 times per query and cannot be cached, because its input is the pair. That is why the shortlist size is the tuning knob: it sets both the reranking cost and the ceiling on quality, since a passage the bi-encoder never retrieved can never be recovered.

The complete script

code

your-project/├── corpus.py      # 36 passages, 8 labeled queries - full listing in step 1├── measure.py     # ndcg_at_k helper├── retrieve.py    # step 2, stage 1 only├── baseline.py    # step 3, NDCG@10 on one query before reranking├── rerank.py      # step 4, stage 2 on one query├── pipeline.py    # step 5, both stages over all queries├── per_query.py   # step 5, the same measurement broken out per query└── timing.py      # step 5, what the reranking stage costs

pipeline.py is listed in full in step 5, where you create it.

Where to go next

Tune the shortlist size. Change TOP_K from 20 to 5, then to 35, and re-run. Reranking cost scales linearly with it, and recall of the correct passage is capped by it. Re-run timing.py at each setting and find where the NDCG gain stops paying for the latency.

Swap in a faster reranker. cross-encoder/ettin-reranker-17m-v1 is smaller than the MiniLM model used here and benchmarks faster on CPU. Change the one model string and re-run pipeline.py - the measurement harness you built is what makes that a two-minute experiment.

Fine-tune the reranker on your own data. The ms-marco checkpoint is general-purpose. A cross-encoder trained on your own query logs will beat it on your corpus, and you now have the harness to prove that rather than take my word for it. See fine-tune a cross-encoder for semantic similarity.

Cross-check your metric against the library's. CrossEncoderRerankingEvaluator computes the same before-and-after NDCG independently. Pass always_rerank_positives=False, or it injects known-relevant passages into the rerank pool even when stage 1 never retrieved them, which inflates the reranked number.

References

Nogueira, R., and Cho, K. (2019). Passage Re-ranking with BERT. arXiv:1901.04085. https://arxiv.org/abs/1901.04085
Thakur, N., Reimers, N., Rucklé, A., Srivastava, A., and Gurevych, I. (2021). BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Models. NeurIPS 2021 Datasets and Benchmarks Track. arXiv:2104.08663. https://arxiv.org/abs/2104.08663
Sentence Transformers. CrossEncoder API reference (5.6.x). https://sbert.net/docs/package_reference/cross_encoder/cross_encoder.html
Sentence Transformers. Migration guide, v5.x to v5.4+. https://sbert.net/docs/migration_guide.html
Sentence Transformers. Retrieve and Re-Rank. https://sbert.net/examples/sentence_transformer/applications/retrieve_rerank/README.html
Hugging Face. cross-encoder/ms-marco-MiniLM-L6-v2 model card. https://huggingface.co/cross-encoder/ms-marco-MiniLM-L6-v2
Hugging Face. sentence-transformers/all-MiniLM-L6-v2 model card. https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2
Hugging Face. Introducing the Ettin Reranker Family (2026-05-19). https://huggingface.co/blog/ettin-reranker
scikit-learn. sklearn.metrics.ndcg_score. https://scikit-learn.org/stable/modules/generated/sklearn.metrics.ndcg_score.html

AI Engineering

More Articles

Reranking for RAG: Boosting Answer Quality in Retrieval-Augmented Generation

Follow for more technical deep dives on AI/ML systems, production engineering, and building real-world applications:

How to Rerank Retrieval Results with a Cross-Encoder

What you'll build

Prerequisites

Bi-encoder vs. cross-encoder: why you need both stages

Step 1: Build a corpus with planted decoys

Step 2: Retrieve candidates with a bi-encoder

Step 3: Measure the baseline with NDCG@10

Where 0.6697 comes from

Step 4: Rerank the candidates with a cross-encoder

Step 5: Measure the reranked pipeline, honestly

What it costs

Fixing the errors you'll hit running this

Fix: TypeError: CrossEncoder.predict() got an unexpected keyword argument 'inputs'

The full two-stage pipeline, end to end

The complete script

Where to go next

References

Get the next article by email

Books by Ranjan Kumar

Building Real-World Agentic AI Systems

The ChatML Handbook

The Chat Templates Handbook

Comments

What you'll build

Prerequisites

Bi-encoder vs. cross-encoder: why you need both stages

Step 1: Build a corpus with planted decoys

Step 2: Retrieve candidates with a bi-encoder

Step 3: Measure the baseline with NDCG@10

Where 0.6697 comes from

Step 4: Rerank the candidates with a cross-encoder

Step 5: Measure the reranked pipeline, honestly

What it costs

Fixing the errors you'll hit running this

Fix: TypeError: CrossEncoder.predict() got an unexpected keyword argument 'inputs'

The full two-stage pipeline, end to end

The complete script

Where to go next

References

Related Articles

Get the next article by email

Books by Ranjan Kumar

Building Real-World Agentic AI Systems

The ChatML Handbook

The Chat Templates Handbook

Comments