Combining different answers.
I started with reciprocal rank fusion: combining the ordered results from several search methods into one ranking. In the IC(R/W)-RRF project, I explored whether agreement between rankers and confidence in individual results could improve that combination.
I built fusion variants and an evaluation harness, using generated ranking outputs and relevance judgments from the TREC Deep Learning 2019 and 2020 collections. The initial comparison used four lexical rankers.
Change the setup. Check the finding.
An early confidence-based variant improved the recorded top-ten ranking metric in the original four-ranker setup. Adding a proximity ranker reversed the comparison, and the original improvement did not carry over to the second collection.
The recorded results also exposed a tradeoff: improving the top-ten metric could coincide with a worse position for the first relevant result. That made the evaluation conditions and the choice of metric part of the story.
Two ways to adapt.
I next explored a mixture that responds to how much the rankings overlap, followed by a small supervised classifier that chooses a fusion method for each query. The first uses observable ranking agreement; the second learns from judged examples.
The saved classifier results improved on one collection, but did not show the same advantage on the other. Training on one collection and applying the selector to the other failed to beat the best fixed comparison in either direction.
What the experiments left open.
The work produced implementations, comparison scripts and a record of promising and unsuccessful experiments. The results remained exploratory: small query sets, selected ranker combinations and evaluation details still needing reconciliation.
A stronger follow-up would separate tuning from final evaluation and test broader collections and ranker combinations. The useful outcome so far was learning which findings survived a changed setup and which needed another explanation.