Split queries improve reranking of instructions in search results
Every Clause Counts: Instruction-Following Reranking by Multiplying Small, Answerable Questions
Information RetrievalComputation and Language
Summary
When people search online using instructions like ‘beginner-friendly’ or ‘official source only,’ search engines often struggle to rank the results correctly. The paper presents a new method that breaks down these instructions into small, clear questions and checks each one separately. Combining the answers gives a better overall ranking of search results that respect the instructions. This approach improves how well the rankings match user expectations without needing extra training.
What this means in practice
- •For search engine engineers: Improve result ranking by separately verifying parts of user instructions to more accurately follow specific query requirements.
- •For customer support platform developers: Enhance automated response ranking by decomposing complex user queries into individual criteria for better compliance checking.
Authors
S. M. Mahbub Uddin Ahmed, Abdullah Al Musaddiq
Abstract
People often attach instructions to their search queries, asking for pages written for beginners, documents in a certain language or official sources only. Most rerankers were trained to judge topical relevance and treat such instructions as a few extra words. On the InfoSearch benchmark, even GPT-4o used as a list-wise reranker reaches an average WISE (Weighted Instruction Sensitivity Evaluation) of only 33.5. We present DMReranker (Decision-Model Reranker), a zero-shot reranker that splits an instructed query into a topic and a short list of typed clauses. A small judge model answers one narrow question per clause, and the answer probabilities are multiplied into a single score. Nothing is trained, and the configuration was frozen before the test runs. Reranking the same top-100 candidates as the published baselines, DMReranker reaches a macro WISE of 50.9 over all six InfoSearch dimensions while keeping original-mode nDCG@10 at 82.9. It trails GPT-4o on the stricter SICR (Strict Instruction Compliance Ratio) metric, 19.2 against 32.1. We show that this gap follows from how a product of probabilities behaves when SICR compares raw scores across runs. Computed from ranks, our SICR is 67.0. On the same judge, splitting the instruction raises WISE from 32.9 to 47.6 over a single combined question, while raw-score SICR falls. Scoring all six dimensions cost about $10 at list price.