Tests show question changes affect retrieval model answers sometimes
The Company a Question Keeps: Auditing Open Jev-Style Models for Retrieval
Information RetrievalComputation and Language
Summary
Sometimes when you change a question or the information shown to a computer model that finds answers, the model's choice changes too. The authors studied several popular models based on Jev's method to see if each question is judged independently. They found that around 9% to 21% of decisions changed when questions or evidence were altered, but often the model's overall correctness stayed similar. They also found technical details like formatting can cause unexpected changes.
What this means in practice
- •For search engine engineers: Identify when question changes cause inconsistent results in retrieval models to improve answer reliability.
- •For ai system maintainers: Detect subtle input format issues that trigger model answer changes to improve system robustness and debugging.
Authors
Jianbo Chen
Abstract
Jev's documented interface evaluates typed questions independently against shared evidence. We audit request independence in seven open Jev-style checkpoints on 276 constructed three-passage states from 92 TREC retrieval queries. Paired interventions separately change requested questions, displayed evidence, and candidate eligibility. Request expansion induces strict binary flips in 9.1%/8.0% of Nimble's judgments and changes 21.4%/18.9% of selection-or-abstention actions in TREC DL2019/2020 under a threshold of 0.5 fixed before inference. Correct-action rates change by 0.0/+8.2 percentage points, respectively, with 95% query-bootstrap intervals of [-6.0,6.0] and [3.8,12.6]. Field masking traces all 70 request-set flips to the schema fields serialized into each scoring prompt. In Von, fixing a shared separator's position eliminates two eligibility-induced preference flips in selected diagnostic states. Laya-typed's default CUDA path produces five passage swaps without strict binary flips, and both yearly action-change intervals include zero; in an FP32 rerun, its probability movement vanishes at the model's four-decimal reporting resolution.