Re-ranking and late interaction improve scientific question answering retrieval
Re-ranking and Late Interaction Drive Retrieval Quality: A Controlled Comparison of RAG Strategies for Scientific Question Answering
Information RetrievalArtificial IntelligenceMachine Learning
Summary
Finding accurate answers to scientific questions often involves searching many research papers. The authors compared six ways to retrieve and rank these papers to help a large language model find the best information. They tested these methods on a large set of recent academic papers and a new set of synthetic questions. Their results highlight that re-ranking retrieved results and using late interaction methods can improve how well the system finds relevant documents. This work also provides tools and data for others to test and build better retrieval systems for science.
What this means in practice
- •For search engine developers: Improve scientific literature search tools by integrating re-ranking and late interaction retrieval methods to boost answer accuracy in question answering systems.
- •For knowledge management teams: Enhance internal document retrieval by adopting advanced retrieval pipelines tested on large academic corpora, enabling better grounding of AI-generated answers.
Authors
Bhagyesh Rathi, Eshan Chawla, William B. Andreopoulos
Abstract
Retrieval-Augmented Generation (RAG) is now the standard way to ground Large Language Models (LLMs) in external knowledge, yet the design space of retrieval pipelines is large and the trade-offs between variants are not well understood, especially on domain-specific corpora at realistic scale. In this work, we present a controlled comparison of six retrieval strategies for scientific question answering: (i) classic top-k dense retrieval, (ii) LLM-based query rephrasing, (iii) query rephrasing followed by LLM-based reranking, (iv) multi-query fusion via Reciprocal Rank Fusion (RRF), (v) an agentic tool-call pipeline in which the generator decides for itself whether to retrieve, and (vi) late-interaction retrieval with ColBERTv2. All six pipelines share the same generator (Meta-Llama/Llama-3.1-8B-Instruct), prompt, and evaluation protocol; the five single-vector pipelines additionally share SPECTER2 embeddings and a Chroma vector store; and all six retrieve from the full corpus of 463,971 arXiv papers dated 2024-2025. To support reproducible, large-scale evaluation, we also release a synthetic question dataset of 19,484 problem-statement and methodology questions generated by Llama-3.1-8B-Instruct from a random sample of 10,000 papers across academic domains (query generation succeeded for 9,742 of them), and every strategy is evaluated on this same query set. We describe the architecture and implementation of each pipeline, release the code and the synthetic question dataset, and evaluate each strategy with an LLM-as-a-judge protocol along multiple quality dimensions, together with direct gold-paper retrieval metrics. The result is an open testbed for studying the cost and quality trade-offs of RAG design choices on a research-literature corpus, and a basis for future work on faithfulness, retrieval robustness, and agentic retrieval.