Large language models often act on false information despite warning signals
Beyond Belief: Evaluating and Mitigating LLM Susceptibility to Misinformation
Computation and LanguageArtificial Intelligence
Summary
Large language models (LLMs) can sometimes be tricked into acting on false information even when they say they don’t trust it. The authors studied five popular LLMs and found that many harmful actions still happen after the models label information as untrustworthy. They also tested ways to reduce these mistakes, such as giving warnings before the task or training the models with examples of false information. The results show it’s hard to stop mistakes without also making the models less helpful overall. The authors suggest checking both error rates and whether models still do their job well when making improvements.
What this means in practice
- •For ai service operators: Improve safety checks in AI tools that use external information by integrating warning prompts to reduce harmful actions.
- •For chatbot developers: Design better training protocols using examples of false information to decrease harmful outputs without overly compromising task success.
Authors
Emran Kanaan, Jiho Lee, Ibrahim Odat, Soroush Vosoughi, Weicheng Ma
Abstract
Tool-using large language models (LLMs) must decide whether external information warrants action. We investigate whether their expressed credibility judgments reliably indicate behavioral resistance to misinformation. We evaluate five LLMs on 196 constructed false notices paired with legitimate counterparts and 204 archived attacks adapted to simulated tool use. When asked to act on the constructed notices, models issue harmful tool calls in 27%--99% of cases they have rated untrustworthy. Removing the preliminary rating step increases harmful execution by 15 percentage points on average, showing that credibility elicitation itself changes the behavior being evaluated. Interventions that improve credibility discrimination do not consistently yield safer actions. A simple forewarning significantly reduces harmful execution in four of five models, but neither tested generated-example prebunk outperforms it in the main evaluation. These reductions can also compromise legitimate task completion. Supervised fine-tuning on generated false notices reduces harmful execution on held-out archived attacks under the rating-first protocol, with model-dependent utility costs and weaker protection without rating elicitation. Our findings motivate evaluating misinformation defenses through both harmful actions and legitimate task completion, while accounting for the behavioral effects of the evaluation protocol.