Large language models often answer wrong when context misleads them
MisleadQA: Diagnosing LLM Failures under Misleading Context
Computation and Language
Summary
Large language models (LLMs) try to answer questions using extra information called context. This paper shows that even if the context is true, it can mislead LLMs if it talks about a different subject than the question. Models get confused more by false context than by true but out-of-scope context. Adding correct context helps with true out-of-scope but less so with false context. The authors created a dataset to test this and show that models sometimes avoid answering when told to be careful.
What this means in practice
- •For llm developers: Test and improve model performance by checking if context applies to the question, not just if it’s true or false.
- •For chatbot builders: Design chatbots that use an abstention option to avoid wrong answers when given misleading context.
Authors
Hui Wang, Bohao Chu, Qianli Wang, Shuning Zhang, Norbert Fuhr
Abstract
Large language models (LLMs) are widely used to answer questions from retrieved or user-provided context. However, provided context can be factually true yet out-of-scope, describing a different entity, time, or location from the one the question targets, and still lead models to a wrong answer. Prior work has mainly studied factually false or irrelevant context. It remains unclear whether such out-of-scope context misleads models as much as false context, and whether adding gold context prevents models from being misled by either. We introduce **MisleadQA**, a diagnostic dataset of multiple-choice questions with an abstention option. Each question pairs gold context (G) with true but out-of-scope context (S) and false but in-scope context (F) supporting the same wrong answer. We evaluate six open-weight LLMs. On questions they answer correctly without context, providing S alone leads them to switch to this wrong answer in 29.3% of cases, and providing F alone does so in 75.4%. Adding G nearly removes the effect of S, but F still misleads models in 21.0% of cases, especially when F comes first. A context-faithful instruction reduces these errors mainly through abstention and lowers accuracy even with G. These findings suggest that evaluations should check whether context applies to the question, not only whether it is true, and should report correct, misled, and abstention rates together. Data and code are available at [https://anonymous.4open.science/r/MisleadQA](https://anonymous.4open.science/r/MisleadQA).