Fast interactive ranking improves document search using AI feedback
Quantum-Inspired Pairwise Ranking Aggregation via Active LLM Feedback
Information RetrievalComputation and Language
Summary
Combining results from several search tools usually takes a lot of work and computing power. The authors introduce a new method that uses AI to compare only the most confusing pairs of documents to decide how to mix scores. This approach makes searching faster and just as good or better than older ways, without needing extra data or training. Their experiments show the method works well on various search tasks with fewer AI calls, saving time and cost.
What this means in practice
- •For search engine developers: Improve document ranking by interactively combining multiple search results using AI only on uncertain pairs, reducing processing and maintaining high accuracy.
- •For content recommendation teams: Enhance recommendation lists by quickly learning to weight multiple scoring methods with minimal AI feedback, leading to better user relevance.
Authors
Quan Mai, Susan Gauch, Gibson Nkhata
Abstract
Ranking aggregation from multiple retrievers is a powerful yet under-explored technique in modern retrieval and RAG systems. We introduce LLM-IRA, the first extension of the quantum-inspired interactive ranking aggregation framework to document reranking. By leveraging large language models (LLMs) solely for pairwise judgments on high-disagreement document pairs, our method learns fusion weights in a principled, interactive way. The final ranking is a simple, ultra-fast pointwise linear fusion of scores, no full pairwise matrix or expensive listwise prompting is required. Experiments on the BEIR benchmark and TREC Deep Learning tracks with an open-weight instruction-tuned judge (Llama-3.1-8B-Instruct) show that LLM-IRA matches or numerically exceeds strong static fusion baselines on 5 of 6 evaluated benchmarks. We report two weight-update variants: momentum blending and a full accumulated-history refit, which together reach statistical significance (Wilcoxon signed-rank, p<0.05) over the best unsupervised aggregator on 4 of 6 benchmarks (DL19 under momentum; DL20, DBpedia, and NFCorpus under accumulated refit, which is the stronger of the two configurations overall). On the benchmarks we profiled (DL19, DL20), it already clears the best unsupervised baseline within as few as 5 LLM calls per query, well before exhausting its typical ~25--30 call budget. LLM-IRA also achieves cost-effectiveness comparable to state-of-the-art zero-shot LLM rerankers at an order of magnitude fewer LLM calls, and requires no training data or model fine-tuning.