Language models often change answers when shown wrong source claims
When Models Defer to Wrong Answers: A Robustness Audit of Source-Attributed Cues in Multiple-Choice QA
Computation and Language
Summary
Sometimes AI systems that answer multiple-choice questions get confused when given extra information about what another source said, especially if that source is wrong. The researchers tested how often these models switch to wrong answers just because of such misleading cues, even when they initially chose the correct one. They found that a particular way of presenting the wrong source information made the models switch answers about 41% of the time, showing that these AI models can be unstable. This means AI might trust incorrect claims too much, even if it previously seemed confident in the right answer.
language modelsmultiple-choice question answeringmisleading cuesanswer instabilitygroundinginstruction-following modelsMMLU-Pro datasetsource attributionvalid-response accuracy
Authors
Manikandan Ravikiran, Siddharth Vohra
Abstract
Language models often receive a question together with a claim about what another source answered. We audit whether such claims destabilize answers in multiple-choice question answering. For each item, we hold one wrong option fixed across misleading conditions and vary the cue template attached to it. We introduce \emph{neutral-conditioned misleading cue adoption rate} (NC-MCAR), which measures switches to that option only on valid cued trials where the same model first selected the gold answer under a neutral prompt. This is a measure of answer instability, not proof that the model knew the answer or that all deference is irrational. We evaluate four instruction-following models on MMLU-Pro and IndicMMLU-Pro in English, Hindi, Bengali, Tamil, and Telugu. Across 220{,}000 outputs, the expert template yields 41.1\% aggregate NC-MCAR, compared with 12.5\% for the majority template. These two conditions use the same wrong option and final instruction. Filler accuracy remains well above expert-wrong accuracy, while correct-cue prompts have high valid-response accuracy. The audit documents answer instability relevant to grounding under the tested forced-choice prompts: a bare, unverified source claim can outweigh an answer that was previously consistent with the task evidence.