Decision models can rival rerankers in text retrieval with trade offs
Can System-1-Style Decision Models Replace Dedicated Rerankers in RAG? A Systems Evaluation
Machine Learning
Summary
Choosing the best information pieces to show a language AI is important but slow when using special sorting tools called rerankers. The authors studied faster systems called system-1-style decision models to see if they could do this job just as well. They found that these decision models can sometimes match rerankers in quality, but they have limits, especially with longer texts. The choice depends on balancing quality, speed, and memory use, and improving existing rerankers might still be the best option.
What this means in practice
- •For machine learning engineers: Optimize text candidate selection in retrieval-augmented systems by testing system-1 models to reduce latency while maintaining quality.
- •For software developers building chatbots: Incorporate tuned rerankers into retrieval pipelines to balance response speed and relevance in conversational AI.
Authors
Qiaomu Li, Ying Xie
Abstract
Retrieval-augmented generation (RAG) systems often use a dedicated reranker to choose which retrieved candidates reach the language model. Reranking improves relevance but adds latency, so teams look for faster alternatives. System-1style decision models make direct, structured predictions instead of generating long responses. Could one provide comparable reranking quality with less inference overhead? We first screen 21 open-weight models on 11 benchmarks, then compare a promising decision model with a dedicated reranker on identical candidate lists. The default decision-model setup fails on many long candidate sets, so scoring only the requests it completes gives a misleading ranking. Limiting candidate text length largely restores reliability and makes some configurations competitive. A frozen evaluation on new retrieval data supports the reliability and ranking results, but confirms the latency requirement for only one of the two selected configurations. Across workloads, which model ranks better changes, and a tuned reranker remains faster and uses far less GPU memory. A decision model can perform competitive reranking, but replacing a reranker requires weighing quality, latency and memory together, and teams should tune the reranker they already run first.