Large language models can hallucinate due to competing query keywords
When the Wrong Key Wins: Understanding and Detecting Hallucinations in LLMs
Computation and Language
Summary
Sometimes large language models give wrong answers even when they have the right information. The authors found that certain important words in a question can strongly influence the model's answers, sometimes leading it to pick incorrect but strongly associated info. They developed a method that changes these key words to see if the model’s answer changes, which helps detect when the model is likely hallucinating. This approach works well across different models and datasets, and helps explain why some detection methods work better in certain cases.
What this means in practice
- •For ai system developers: Detect hallucinations in AI-generated answers by testing sensitivity to key question words.
- •For content moderation teams: Improve review tools to flag AI outputs that may be based on misleading keywords in queries.
Authors
Xuhan Tong, Jiawei Zhang
Abstract
Large language models can hallucinate even when the knowledge required for a correct answer is already available. We study this failure through a latent-key view of inference, where answer selection depends on competition among associations acquired during pretraining. We show that model predictions can be highly sensitive to individual query keywords, that these influential keywords exhibit entity-specific binding, and that their effects are systematically shaped by pretraining frequency. Multiple bindings can also compete and exhibit higher-order interactions within the same query. Based on this mechanism, we introduce a two-stage keyword-perturbation method for hallucination detection. By removing influential keywords and measuring how the model reorganizes its prediction, the method distinguishes errors caused by misleading key associations from correct decisions supported by diagnostic evidence. Across multiple models and benchmarks, perturbation provides a strong and transferable detection signal, reaching $.910$ AUROC on probe-known ScientistQA. Finally, we extend the same probabilistic framework to four hallucination regimes: knowledge deficit, wrong knowledge, context distraction, and unstable inference. Their operational distributions across benchmarks provide diagnostic context for why different detector families succeed in different settings.