Retriever upgrades cause some query results to get worse unexpectedly
RAGFlip: Measuring Query-Level Negative Flips in Retriever Upgrades
Information Retrieval
Summary
Retriever tools help find relevant passages for search queries, and upgrades aim to improve them overall. But the authors found that some queries that were correctly answered by the old tool stop being answered correctly after upgrading. They call these cases negative flips. Their study showed these negative flips happen often, depending on the dataset and settings. They also showed that combining old and new results can reduce these mistakes.
What this means in practice
- •For search engine developers: Identify and reduce queries for which retriever upgrades cause loss of relevant results, improving search consistency across updates.
- •For digital library managers: Use combined retrieval methods to maintain recall of relevant documents when upgrading search systems.
Authors
Elyas Irankhah, Muhammad Arif
Abstract
Retriever upgrades are typically evaluated using aggregate metrics, which can hide regressions on queries the previous retriever already served correctly. We study these regressions as negative flips: queries for which BM25 retrieves a judged relevant passage and the replacement does not. We evaluate BGE-large, E5-large-v2, and SPLADE on three BEIR collections: Natural Questions, HotpotQA, and FiQA, across five retrieval depths. All replacements improve overall retrieval coverage. Negative flips occur in every setting and vary substantially by corpus, retriever, and depth. At k=1, 8.6-37.5% of BM25 successes are lost across the evaluated settings. Negative-flip rates are lower in the larger-depth settings, where the BM25-supported cohort is defined separately at each depth. These rates use the any-relevant support label. On HotpotQA at k=10, requiring every positive qrel passage raises the negative-flip rate to 12.3-17.3%. Simple fixed-budget combinations with BM25 reduce these regressions, and a HotpotQA reader experiment provides a limited downstream check in which some retrieval flips are accompanied by answer regressions. These results motivate evaluating retriever updates using query-level compatibility alongside aggregate retrieval quality.