Summary
Attention-based rerankers help computers decide which documents best answer a question by looking at how much attention the question pays to each part of the document. They use a trick called calibration to avoid biases, but this trick assumes the calibration step doesn’t accidentally remove useful information. The authors found that when prompts include detailed instructions or personas, this trick can mistakenly ignore relevant signals and hurt performance. They fixed this by adjusting how calibration handles prompts, which improved results especially when prompts have lots of instructions. They also found that example demonstrations in prompts don’t cause this problem because they don’t affect the calibration step.
What this means in practice
- •For search engine developers: Improve document ranking quality on tasks with detailed user instructions by using interpolated null calibration in attention-based rerankers.
- •For chatbot system builders: Enhance chatbot response ranking when prompts include complex instructions or personas by adjusting calibration methods to avoid removing relevant signals.
Authors
Petros Karypis, Hossein Rajaby Faghihi, Peter Chen, Rui Zhu, Noveen Sachdeva, Yan Zhu, Julian McAuley
Abstract
Attention-based rerankers score documents by aggregating query-to-document attention and subtracting a null-query calibration pass to remove positional and structural bias. Although widely used, this calibration assumes that the null pass removes irrelevant signal from each document. We show that modern prompt content, e.g. constraints, instructions, personas, and demonstrations can violate this assumption when it enters the scoring readout, making the null pass relevance-aware rather than null. We find that calibration is especially harmful when applied to prompts containing longer, more detailed instructions as the null-pass step removes relevant signal. Based on these findings, we propose interpolated null calibration, a training-free modification that controls how much of the instruction content enters the null baseline. It recovers attention-based reranking performance on instruction-heavy tasks where standard calibration fails, while preserving calibration's benefits when the null pass remains relevance-agnostic. On instruction heavy tasks, the recovered rankings surpass generative rerankers. We also show that in-context demonstrations improve attention-based reranking with little calibration interference, since demonstrations act only through the query pass and leave the null pass unchanged.