Language models often switch to wrong answers when given misleading source claims

When Models Defer to Wrong Answers: A Robustness Audit of Source-Attributed Cues in Multiple-Choice QA

Computation and Language

Summary

This paper shows that language models answering multiple-choice questions can be easily influenced by false claims about what another source answered. The authors tested models by giving them wrong answer options paired with different misleading prompts and found that models often changed their correct answers to wrong ones when those prompts suggested so. This effect was seen in multiple languages and across many examples, showing that models’ answers can be unstable and overly influenced by unverified source claims. However, the paper clarifies this doesn’t prove the models actually knew the right answers or that all such changes are unreasonable.

What this means in practice

  • For ai system builders: Identify prompt designs that cause language models to shift to incorrect answers when influenced by misleading source claims.
  • For multilingual chatbot developers: Improve reliability of chatbot answers by recognizing when source-attributed claims might destabilize responses in multiple languages.

Authors

Manikandan Ravikiran, Siddharth Vohra

Abstract

Language models often receive a question together with a claim about what another source answered. We audit whether such claims destabilize answers in multiple-choice question answering. For each item, we hold one wrong option fixed across misleading conditions and vary the cue template attached to it. We introduce \emph{neutral-conditioned misleading cue adoption rate} (NC-MCAR), which measures switches to that option only on valid cued trials where the same model first selected the gold answer under a neutral prompt. This is a measure of answer instability, not proof that the model knew the answer or that all deference is irrational. We evaluate four instruction-following models on MMLU-Pro and IndicMMLU-Pro in English, Hindi, Bengali, Tamil, and Telugu. Across 220{,}000 outputs, the expert template yields 41.1\% aggregate NC-MCAR, compared with 12.5\% for the majority template. These two conditions use the same wrong option and final instruction. Filler accuracy remains well above expert-wrong accuracy, while correct-cue prompts have high valid-response accuracy. The audit documents answer instability relevant to grounding under the tested forced-choice prompts: a bare, unverified source claim can outweigh an answer that was previously consistent with the task evidence.
View PDFOpen arXiv