The mechanism
Reciprocal rank fusion
A ranked list is simply a set of results in order. Reciprocal rank fusion gives each document a contribution from every list in which it appears. A higher position contributes a little more. Those contributions form one combined order.
Follow the same letter across the lists below. Then change the third list and see what moves.
An illustrative calculation
Follow one document through the lists.
Each search method returns an ordered list. Give a document credit for its position in each list, then combine those contributions into one score.
Each method puts a different document first. B appears second in all three.
Search results
Method 1
- Document A
- Document B
- Document D
- Document C
- Document E
- Document F
Search results
Method 2
- Document C
- Document B
- Document A
- Document E
- Document F
- Document D
Search results
Method 3
- Document D
- Document B
- Document C
- Document F
- Document A
- Document E
Reciprocal rank fusion
One combined list
Document B is highlighted through the three source lists and the combined result.
Document B
One contribution from each list
Method 1Rank 2
1 / (60 + 2)
0.016129
Method 2Rank 2
1 / (60 + 2)
0.016129
Method 3Rank 2
1 / (60 + 2)
0.016129
Add the three contributions, then divide by three.
(0.016129 + 0.016129 + 0.016129) ÷ 3= 0.016129
Document B is number 1 in the combined list. This score follows from the supplied ranks; it does not tell us whether the document is relevant.
The rule: each list contributes 1 / (60 + rank). The offset of 60 softens the difference between nearby positions. Rank begins at 1, so first place contributes 1 / 61. Average the contributions across the lists. This matches the baseline’s equal weighting; summing instead gives the same order.
A–F are invented document labels. These are illustrative rankings, not research queries or saved evaluation results. Values are rounded for display; the ordering uses unrounded scores. Exact ties use alphabetical order in this example.
The hypothesis
The confidence hypothesis
Position is a useful starting point. I wanted to test whether the combination could also respond to confidence in a particular result, agreement among search methods, and differences between queries.
I built adaptive variants and compared them with ordinary RRF. The first setup combined four lexical search methods: BM25, tuned BM25, TF-IDF and query likelihood. They produced their rankings over shared candidate passages.
When does listening differently
lead to a better ordering?
The experiment
Results under changed conditions
The confidence variant improved the recorded top-ten measure in the original four-method setup. Adding a fifth method, which considers how close query terms appear to one another, reversed the comparison. The early advantage also did not carry over to the second collection.
The measure here, NDCG@10, rewards relevant documents nearer the top of the first ten results. Each setting has its own comparison.
NDCG@10 · higher is better · axis detail: 0.34–0.47 · full 0–1 scale
The original four
2019 · 43 judged queries
Add a fifth perspective
2019 · the same 43 queries
Original four, new collection
2020 · 54 judged queries
Saved exploratory results from project-generated rankings on TREC Deep Learning collections. These are aggregate ranking scores under the project’s evaluation method, not percentages of correct answers. The phone bars use the full 0–1 scale.
Read the values and all four methods +
| Setting | Ordinary RRF | Confidence v5 | Mixture v6 | Selector v7 |
|---|---|---|---|---|
| The original four2019 · 43 judged queries | 0.3645 | 0.3800 | 0.3832 | 0.3785 |
| Add a fifth perspective2019 · the same 43 queries | 0.4031 | 0.3698 | 0.3962 | Not reported |
| Original four, new collection2020 · 54 judged queries | 0.4483 | 0.4373 | 0.4393 | 0.4644 |
“Better” depends on
what you measure.
In the initial 2019 comparison, the confidence variant raised the top-ten score. The reciprocal-rank measure, which rewards finding the first relevant result sooner, fell.
- Top-ten ranking · NDCG@10
- 0.3645 0.3800
- First relevant result · reciprocal rank
- 0.7308 0.7108
Ordinary RRF → confidence v5. Saved aggregate results for the original four-ranker, 43-query setting.
The next experiments
Adaptive fusion methods
The changed result opened two directions: respond directly to agreement between rankings, or learn which method to choose from examples with judged relevance.
v6 / Agreement-dependent mixture
Agreement between rankings
For each query, inspect how much the methods’ top 30 results overlap. Use that agreement, along with differences in rank, to adjust the combination.
Agreement is observable. It does not, by itself, establish that a result is relevant.
v7 / Trained selector
A trained method selector
Learn from judged examples which method works better for a query. A small classifier then uses rank and score features to choose ordinary RRF or the confidence variant.
Training uses relevance judgments. A new query is routed using its ranking features.
The transfer test
Transfer between collections
The saved selector exceeded all three fixed alternatives within the 2020 collection. In 2019 it beat ordinary RRF but trailed the confidence variant and the mixture. Training on one collection and applying it to the other did not beat the best fixed comparison in either direction.
0.4412 selector vs. 0.4483 ordinary RRF
0.3800 selector vs. 0.3832 mixture v6
Saved NDCG@10. These comparisons describe two tested transfers, not a limit on what future methods could achieve.
Where the work stands
Current checkpoint
This is a continuing investigation, now paused at the checkpoint after cycle 23. I test a hypothesis, inspect what changed, and use that evidence to decide what to test next.
The adaptive results above remain exploratory. Those collections contain 43 and 54 judged queries, use related candidate data and project-generated rankers, and have not been independently reproduced here. Tuning and evaluation were not fully separated.
The later cycles separated them on SciFact, where the systems were fixed before the collection was first measured, and then worked on 30 reused TREC-COVID topics as a disclosed development panel. The work now runs as numbered research cycles, twenty-three so far. The experimental ones commit their protocol, comparisons and stop rules before any outcome is read. They added a fifth source, SPLADE++, a ranker trained to weight words, used as saved rankings, and replaced significance tests with bounds that hold under every completion of the missing relevance judgments. Cycle 05 hit an instrument failure and cycle 22 stopped on a checksum mismatch. Both reports stay in the record, as does cycle 20, where the project’s own controlled test excluded a proposed advantage on the 30 topics tested.
The latest cycle used every eligible team’s automatic submissions to the same 30 topics: 81 runs from 29 teams. Averaging the rankings of a team’s runs and adding one fixed neural reranking run did better than picking one of that team’s runs at random with the same partner, by between 0.24 and 6.35 points of top-ten precision, however the missing judgments turn out. That gain is an average across teams. It holds for 13 teams, reverses for 2 and is unsettled for the rest. These are reused development topics, and 78 of the 81 archive files do not match the checksums listed beside them, so their identity with the original submissions is unverified. A test on fresh topics is the next question the record names, and it has not been designed.
Of 29 teams, 13 have an interval above zero, 2 below zero, and 14 cross or touch zero. Pooled over all teams the interval is +0.24 to +6.35 P@10 points.
Grouped output minus a random member, same partner · P@10 points
Each row is one team, sorted by its lower bound: its grouped output against one of its own runs chosen at random, both combined with the same fixed partner run. The bounds are logical bounds over every completion of the missing labels, not confidence intervals, on 30 reused topics. Rows are unlabeled; the table below gives every interval.
Read the 29 intervals +
| Row | Lower | Upper | Sign |
|---|---|---|---|
| 1 | −5.78 | +24.78 | Not settled |
| 2 | −3.17 | +17.83 | Not settled |
| 3 | −3.00 | +0.67 | Not settled |
| 4 | −2.78 | −1.78 | Negative |
| 5 | −1.33 | +4.89 | Not settled |
| 6 | −1.33 | +7.83 | Not settled |
| 7 | −1.17 | +0.33 | Not settled |
| 8 | −0.89 | 0.00 | Zero or below |
| 9 | −0.78 | −0.44 | Negative |
| 10 | −0.44 | +0.11 | Not settled |
| 11 | −0.44 | +15.44 | Not settled |
| 12 | −0.33 | +1.50 | Not settled |
| 13 | −0.22 | +0.67 | Not settled |
| 14 | −0.22 | +2.00 | Not settled |
| 15 | −0.11 | +20.00 | Not settled |
| 16 | 0.00 | +1.67 | Zero or above |
| 17 | +0.33 | +0.67 | Positive |
| 18 | +1.11 | +4.00 | Positive |
| 19 | +1.22 | +4.00 | Positive |
| 20 | +1.22 | +10.33 | Positive |
| 21 | +1.44 | +3.78 | Positive |
| 22 | +1.67 | +7.11 | Positive |
| 23 | +2.00 | +2.67 | Positive |
| 24 | +2.11 | +4.56 | Positive |
| 25 | +2.11 | +21.67 | Positive |
| 26 | +2.17 | +4.33 | Positive |
| 27 | +2.67 | +6.78 | Positive |
| 28 | +3.11 | +6.00 | Positive |
| 29 | +3.78 | +16.89 | Positive |
Inside the evaluation record +
What was measured
The project’s harness computes ranking metrics over generated lexical runs and stored relevance judgments. These are saved project results, rather than official submitted systems or a new evaluation run for this page.
What needs reconciliation
The statistical helper uses a normal-tail approximation for a statistic described as a paired t-test. The selector’s reported p = 0.003 compares it with v5; the reported comparison with ordinary RRF is p = 0.125. This record does not establish a general statistically significant improvement over RRF.
Configurations were selected using these same collections. An independent outer evaluation with tuning confined inside each training fold is not shown. A routing-count tally is internally inconsistent, so no routing percentages are presented. The classifier implements nine features, while the specification describes ten.
What follows from that
The comparisons motivate further experiments. They do not establish a universal improvement, a minimum training-set size, an impossibility of transfer, or production effectiveness.
Off the benchmark
Two real memories.
One lesson about rules.
In October my agent team asked the same question of two collections it actually searches: its own memory, the reports and records its agents write as they work, and a second, client-domain corpus. Each has about twenty questions whose answers we already know, 19 and 20. Both search with a list that matches meaning, using embeddings, and a list that matches words, using BM25; the client-domain search adds a third, page-level list. The round reused the research code without changing it. It was not one of the numbered cycles: nothing was registered before the scores were read.
The first result was about the test. Our first score on the client-domain corpus was 0.604; the corpus’s own harness reports 0.743. Two evaluation rules explained the whole gap. Counting a result as correct when its address contains the known answer, rather than ends with it, moved the score by 0.22. Fusing whole pages first, rather than passages, moved it by 0.06. With both rules matched, the run reproduced the harness exactly, and no fusion variant moved the score by more than 0.05. A comparison of fusion methods has to state both rules, or it is partly measuring the rules.
Change in MRR@10 · same questions, same lists · bars run from zero; an outlined bar to the left is a loss
- Evaluation rules, changed one at a timeCorrect when the address contains the answer, not ends with it+0.222
- Fuse whole pages first, not passages+0.061
- Fusion variants against ordinary RRFRRF, k = 10+0.050
- Consensus weights (v5 without scores)+0.040
- RRF, k = 30+0.036
- Mixture v6, forced to fire−0.008
- Mixture v6 (never fired; equal list weights)+0.003
Twenty questions with known answers on a client-domain corpus, ordinary RRF unless named. Each rule bar changes that rule alone; each variant bar compares with ordinary RRF under the rules that reproduce the corpus’s own harness. Each variant’s change comes from one to five of the 20 questions moving.
Read every variant on both corpora +
| Variant | The team’s memory19 questions, first labels | A client-domain corpus20 questions, substring gold (lenient) |
|---|---|---|
| Ordinary RRF, k = 60 | 11 · 16 of 19 · 0.706 | 12 · 18 of 20 · 0.743 |
| RRF, k = 30 | 11 · 15 of 19 · 0.702 | 13 · 18 of 20 · 0.779 |
| RRF, k = 10 | 11 · 15 of 19 · 0.694 | 14 · 18 of 20 · 0.793 |
| Confidence v5The client-domain lists carry ranks only, so v5 falls back to its consensus weights there. | 8 · 14 of 19 · 0.610 | 14 · 18 of 20 · 0.783 |
| Mixture v6Never fired on either corpus. Its one changed question on the client-domain corpus comes from the research code’s unweighted RRF base. | 11 · 16 of 19 · 0.706 | 12 · 18 of 20 · 0.746 |
| Mixture v6, forced to fire | 11 · 15 of 19 · 0.702 | 12 · 18 of 20 · 0.735 |
| Best single choice per questionAn oracle that picks the meaning list, the word list or ordinary RRF for each question with the answer in hand. Not recomputed for the client-domain corpus under the aligned rules. | 12 · 16 of 19 · 0.750 | Not computed |
The mixture stayed off.
Mixture v6 adjusts the combination only when the lists’ top 30 results overlap by at least 0.35. Here they barely overlapped: 0.03 to 0.25 on the team’s memory, and a median of 0.14 on the client-domain corpus, where none of the 20 questions reached 0.35. So v6 reduced to ordinary RRF, which is what it was built to do when methods disagree this much. Its +0.003 on the client-domain corpus is one question, and it comes from the research code weighting the three lists equally, not from the mixture. Forced to act on the team’s memory, it changed seven questions, mostly for the worse.
k did not travel.
A smaller RRF constant helped one corpus and hurt the other. At k = 10 the client-domain corpus gained 0.05 and the team’s memory lost 0.012. That is the shape of the TREC transfer above: a gain on one collection, the opposite sign on the next.
Confidence cut both ways, on different tests.
On the team’s memory, confidence v5 lowered the score by 0.10 and first-place answers from 11 to 8. The client-domain lists carry ranks only, so there v5 falls back to its consensus weights, which gained 0.04: three questions up, two down. Different variants on different collections, a few questions each. No claim either way.
What it adds up to
Inconclusive, and
worth the next loop.
With 19 and 20 questions, the largest movements are two to four questions. No variant beat ordinary RRF reliably on either corpus. That is consistent with the TREC result; it does not prove it. The team’s memory search stays on ordinary RRF with k = 60.
There is room above it. Choosing the best of the meaning list, the word list and ordinary RRF separately for each question would have scored 0.044 higher on the team’s memory, 0.750 against 0.706. That choice needs a selector that learns from judged questions, and 19 are too few to train one and test it. The team’s search already lets an agent mark each answer as a hit or a miss. That log grows with ordinary work, and it is the next loop: once it holds enough judged questions, the selector can be tested on real memory.
Under a second labeling of the same 19 questions, every variant moved in the same direction.
Engineering
Experiment design and evaluation
This is offline research on public search benchmarks. A benchmark gives you questions, candidate documents and judgments of which documents answer which question, and some of those judgments are always missing. The engineering here is what it takes to get a trustworthy answer out of that record. In the experimental cycles I decide the comparison before any score is seen, check the scoring code against the official tool, and work out exactly how far a missing judgment could move the verdict. It runs in none of my products. I did try it once, offline, over the knowledge graph behind Ascend: four RRF fusions among eleven ways of ranking its nodes, scored against blind-graded questions. No pair differed significantly, so the live index stayed as it was.
- Protocol
23 numbered cycles · protocol committed first · stop rules · hash-frozen sample frame · failures kept
Each experimental cycle commits what will be compared, which metric decides and when to stop, before any outcome is read.
How it works
Each experimental cycle commits what will be compared, which metric decides and when to stop, before any outcome is read; the two that could not, a retrospective inspection and a post hoc descriptive cycle, say so in their own reports. For the TREC-COVID census, a run over every eligible team’s archived submissions, the set of runs was fixed by a rule frozen before any ranking was downloaded: the 29 teams with at least two automatic submissions, after excluding two teams the project had already used, 81 runs out of 143. The record keeps what stopped as well as what finished: an instrument failure, a data-decoding failure on a first SciFact attempt, and a checksum mismatch that halted cycle 22.
- Fusion
RRF k=60 · one-based ranks · math.fsum · Fraction fallback · lcm(61..160) integer kernel · ties by document ID
Every variant starts from one rule: a document scores the sum, over the lists it appears in, of one over sixty plus its rank, with ranks counted from one.
How it works
The contract written for the numbered cycles sums with fsum, falls back to exact rationals if a float overflows, and breaks equal scores by document ID, so rounding and input order cannot decide a place. The earlier harness broke ties by input order, and cycle 01 records the small shift in one saved baseline when that changed. The census cycles score in integers, scaling each term by the least common multiple of 61 through 160, so near-ties are settled exactly.
- Sources
7 lexical rankers from scratch · BM25 k1=1.2 b=0.75 · QL Dirichlet μ=2000 · char 4-gram random projection · SPLADE++ cached runs · SHA-256 pinned inputs
The rankers are written from scratch over MS MARCO top-1000 candidates, and the SPLADE++ rankings are cached runs acquired against a pinned SHA-256.
How it works
The rankers are written from scratch over MS MARCO top-1000 candidates: BM25 (k1 1.2, b 0.75) and a tuned BM25, TF-IDF, query likelihood with Dirichlet smoothing (μ 2000), bigram TF-IDF, term proximity and a 256-component random projection of character 4-grams, each keeping its top 200. The SPLADE++ rankings, from a learned-sparse model trained to weight terms, are cached runs, acquired against a pinned SHA-256 with 1,000 candidates per query. No encoder ran here, and the project builds no embeddings or vector index. SciFact’s inputs, a gigabyte across five files, were fetched once and checked against pinned identities before use.
- Data identity
publisher checksum compared · stopped attempt preserved · SHA-256 of current archive bytes · every original digest kept · depth cap fixed in advance
Every archived run’s bytes are compared with the checksum its publisher lists. Under the first contract a mismatch stopped the run before evaluation, and that stopped attempt stays in the record.
How it works
A separate contract then identified the current bytes at all 81 official archive URLs, kept every original digest, and reports all 78 mismatches rather than dropping a run. That covered 93,596,624 bytes, 1,979,714 rows and all 2,430 member-query lists, including 244 shorter than the 100-document cap, which was fixed in advance. No label was read before the policies were saved.
- Metrics
nDCG@10 · MAP@100 · reciprocal rank · standard-library Python · NIST trec_eval gate · max diff 0.000035
Before the SciFact run, the local binary nDCG@10 was gated against the official NIST trec_eval built at a pinned revision.
How it works
The harness is standard-library Python with the metrics written out: nDCG with exponential gain and the ideal taken from every judged grade, MAP@100 over all relevant documents, and reciprocal rank. Across 11 synthetic queries the largest difference from its four-decimal output was 0.000035, inside a tolerance of 0.00005 frozen in advance.
- Missing judgments
sharp bounds · every label completion · not confidence intervals · top-10 RBP p=0.8 · width removed per label · 1,070 later labels
Each missing grade is allowed to take every value it could, the same value on both sides, and the comparison reports the exact lowest and highest mean difference that results.
How it works
Benchmarks judge only some of the documents a system returns. Here an unjudged document is not scored as irrelevant: each missing grade is allowed to take every value it could, the same value on both sides, and the comparison reports the exact lowest and highest mean difference that results. Adding SPLADE++ to four-list RRF came out ahead under every completion: top-10 rank-biased precision (persistence 0.8) between +0.041 and +0.096 on the 43 queries of 2019 and between +0.037 and +0.067 on the 54 of 2020. They are logical bounds on a fixed panel. No confidence interval is involved. Revealing one missing label removes exactly its coefficient of interval width, so unjudged documents can be ranked by how much they would settle. On the 2019 queries, one label per query chosen that way was projected to narrow the interval from 0.0552 to 0.0267, against 0.0325 for labels picked uniformly. On TREC-COVID the same reasoning predicted which comparison new labels would narrow. Then 1,070 later official labels inside the saved candidate scope were revealed, with no ranking touched. The predicted narrowing occurred, and the two-run hybrid’s lead over its lexical source became at least 4 points of precision in the top ten under every completion.
- Verification
351 unittest functions · independent reconstruction · 1,669,294 ranking rows · max diff 2.22e-16 · Fraction checker
Separately written checker code rebuilds each cycle’s results, and for SciFact it reconstructed all 1,669,294 ranking rows with exact order and scores.
How it works
Separately written checker code rebuilds each cycle’s results. For SciFact it reconstructed all 1,669,294 ranking rows across four lexical sources and five arms with exact order and scores. The largest per-query metric difference was 2.22e-16, and all five official trec_eval exports matched at four decimals. For the census, a separate checker with its own parsers and exact rational arithmetic rebuilt all 3,300 ordered result heads and every source, policy, analysis and provenance field, and a further read-only audit re-summed the coefficient ledger to confirm both interval endpoints. The research suite defines 351 test functions across 29 files.
Mean binary nDCG@10 · 300 queries · higher is better · axis 0.66 to 0.71, not from zero
- A Four lexical lists, RRF0.6673
- B Four lexical lists + SPLADE++, RRF0.6867
- S SPLADE++ alone0.7036
- H BM25 + SPLADE++, RRF0.7054
B − A, the contrast fixed before the run: +0.0193 · 47 queries better, 241 tied, 12 worse
Saved result of one cycle on BEIR SciFact, all 300 test queries, under a protocol frozen the cycle before. Mean binary nDCG@10 with missing judgments scored as zero; no significance test. The SPLADE++ rankings are cached runs, so no model ran here. Adding SPLADE++ to the four lexical lists gave the predicted gain, and the plain two-list hybrid still finished ahead of it.
Limits. This is research and serves no product. The trec_eval gate covered binary nDCG@10 on synthetic queries, and the census heads were not exported through it. The bounds are logical bounds over a fixed panel of queries. They cover neither future queries nor assessor error. The harness’s p-values use a normal tail on 43 and 54 queries, and none is a headline result. The TREC-COVID label check was post hoc on 30 reused topics, and 78 of that census’s 81 archive files did not match their listed MD5, so it describes the archive bytes as downloaded. The v5 to v7 tables are saved results, not reruns. Reconstruction was done by separate code inside the project, and no third party has replicated it.