Ranking methods differ in preference and position accuracy in search results
Preference Is Not Position: Evaluating Reranking on NevIR in Fixed Hybrid Candidate Pools
Information Retrieval
Summary
Sometimes when a search engine finds two very similar text passages, one might be better because it answers the question more accurately, even if it appears lower on the list. The authors studied how ranking methods sort these pairs and found that improving the model to choose the better passage doesn’t always put it near the top. They tested ways to balance picking the better passage and putting it high in the results, showing that combining methods can improve both accuracy and ranking position.
What this means in practice
- •For search engine developers: Improve search result rankings by jointly optimizing passage preference and position for better user experience.
- •For content recommendation teams: Enhance recommendation accuracy by using pairwise ranking methods to better distinguish closely related content.
Authors
Yuning Bian, Lele Xu, Yifan Yuan, Xinyu Yin, Chengyan Zhan
Abstract
Near-identical passages can differ in a decisive condition, such as negation. A ranker may order such a pair correctly while placing the preferred passage far down the retrieved list. We examine this distinction using official NevIR pairs jointly covered by fixed candidate pools from dense, learned sparse, and BM25 retrieval. On 2,736 Test query directions, a pairwise-trained ranker improves strict per-query preference over fixed fusion (56.62% versus 52.30%) while reducing preferred-passage mean reciprocal rank (MRR) from 0.5764 to 0.3055. An exploratory fixed-model replay of a variant with background sampling and revised supervision yields MRR of 0.5859 and strict preference of 54.53%. Bounded pointwise reranking of the fusion top 20 improves both outcomes. Qwen3-14B-AWQ with fusion tie-breaking reaches 77.45% strict preference, 58.99% NevIR-style paired accuracy across 1,363 complete pairs, and preferred-passage MRR of 0.7509. A BGE cross-encoder reaches 69.77% strict preference and MRR of 0.7404. A post-hoc analysis finds that the branch where equal Qwen grades for the two designated passages trigger fusion tie-breaking accounts for 27% of evaluated directions and contains most remaining strict-preference failures. These findings support joint evaluation of pairwise preference, designated-passage position, and computational cost, while distinguishing model judgments from the combined ranking procedure.