Gating skips reranker calls for faster multi-hop retrieval with small accuracy loss
Per-Query Gating of LLM Rerankers for Multi-Hop Retrieval
Information RetrievalArtificial IntelligenceComputation and LanguageMachine Learning
Summary
Large language model (LLM) rerankers improve search accuracy but slow down the system and add cost. The authors develop a gate that decides for each query whether the reranker is needed, based on quick-to-get features before running the costly reranker. This gate skips about half of reranker calls, saving time and money with only a small drop in accuracy. The authors carefully measure harm caused when the gate skips useful reranking and adjust thresholds to control this risk. The gate’s predictive power is limited, so some harmful skips remain despite calibration.
What this means in practice
- •For search infrastructure engineers: Reduce the computational cost and latency of multi-hop document retrieval systems by skipping unnecessary LLM reranker calls per query using a learned gate.
- •For digital assistant developers: Optimize response time and cost by selectively activating LLM reranking only when it likely improves multi-step question answering accuracy.
Authors
Andre Bacellar
Abstract
LLM rerankers add of the order of \$0.2-0.3 per 1,000 queries and about a second of tail latency on top of a graph-augmented dense pipeline such as HippoRAG2, and on three multi-hop benchmarks they improve final-hop top-K coverage on seven of nine (dataset, K) cells, by up to +34.8 pp. We ask whether a learned per-query gate can skip the reranker where it will not help, using only features available before the LLM call (27 score and lexical statistics of the two retrieval lists plus a PCA of a small query embedding) with an executable fallback. Every choice, including the fallback and the threshold, is made inside the training fold and applied once to held-out queries, and harmful skips (the rerank would have found the target, the fallback did not) are reported next to the aggregate coverage. Across nine cells on 2WikiMultiHopQA, MuSiQue and HotpotQA the gate skips 51% of calls at an average held-out LastHop@K cost of 1.2 pp; four cells meet a pre-registered 1 pp rule, harmful skips occur in eight (190 harmful against 136 beneficial), and a random gate at the same skip rate loses 2 to 11 pp on the high-lift cells. A second rule sets each cell's threshold from a pre-specified budget on the expected harmful-skip rate over Platt-calibrated harm probabilities (ECE 0.025 after calibration, 0.094 before): at a 1 pp budget the gate skips 42% at -0.8 pp with 66 harmful skips and six cells within 1 pp, but realised harm exceeds the promise in six cells (mean 1.45 vs 0.83 pp), a selection optimism we quantify; a 0.5 pp budget realises about 1 pp. The harm probabilities are calibrated but barely discriminative (AUC 0.16 to 0.70). An earlier version reported 73% "lossless" savings; that figure rested on an oracle fallback and a wrong MuSiQue target, and we document both.