Benchmark helps pick best query rewrite to improve search results
Route Me If You Can: A Benchmark for Query Reformulation Selection
Information Retrieval
Summary
Sometimes when you search for something, changing the wording of your question can help find better answers. But there’s no one best way to change questions that works every time or for every search engine. The authors made a big test set called QueryRoute that includes lots of original questions and many ways to rewrite them, along with results from different search tools. They use this to see which rewritten question works best for each search, showing there’s room to improve how we pick the best rewrite. This can help future systems choose better questions to get better search answers without redoing all the hard work every time.
What this means in practice
- •For search engine developers: Test different ways to pick the best query rewrite for each search to improve retrieval accuracy without costly reprocessing.
- •For chatbot builders: Improve chatbot search responses by selecting the most effective query rewriting from multiple candidates before retrieving answers.
Authors
Hai Son Le, Negar Arabzadeh, Amin Bigdeli, Radin Hamidi Rad, Sajad Ebrahimi, Charles L. A. Clarke, Ebrahim Bagheri
Abstract
LLM-based query reformulation can improve retrieval, but no single reformulation strategy is consistently optimal across queries, domains, retrievers, or model backbones. This creates an inference-time decision problem: ``Given an original query and a pool of candidate reformulations, which one should be issued to the retriever?''. Existing studies are hard to compare because they use different reformulator pools, retrievers, relevance signals, training labels, and evaluation metrics. We introduce QueryRoute, a benchmark that freezes the expensive artifacts needed to study this decision reproducibly: original queries, generated variants, ranked lists under multiple retrievers, retrieval scores, and per-query oracle labels. The benchmark contains 3,757 queries, 11 candidate systems, five reformulator backbones, and three retrievers across TREC DL, BEIR, and BRIGHT, yielding 619,905 retrieval outcomes. We benchmark supervised classification, routing, QPP, and LLM-as-judge selectors. Results show substantial oracle headroom over fixed reformulators, but current selectors recover only part of it; selector rankings change across retrievers, and similar mean effectiveness can hide different query-level behavior. The released artifacts and evaluation harness allow future selectors to be compared without regenerating variants, rerunning retrieval, or rebuilding judge pipelines. Code and data are available at https://github.com/haisonle001/QueryRoute