Large language models struggle to correct initial false information
PHRBench: A Behavioral Evaluation of Post-Hallucination Reasoning in LLMs
Computation and LanguageArtificial Intelligence
Summary
Sometimes language models start with wrong facts in a conversation, which can lead to mistakes later on when they try to explain or answer questions. The authors created a test called PHRBench to see how well different large language models fix those wrong facts while reasoning. They found that correcting these initial errors is rare, but when it happens, it usually involves the model changing its beliefs more often during the explanation. They also showed that certain details about the wrong facts can help predict whether the model will fix its mistake.
What this means in practice
- •For ai product developers: Improve multi-step AI applications by identifying when language models can successfully correct earlier factual errors during reasoning.
- •For chatbot designers: Enhance chatbot reliability by evaluating and predicting when the model will recover from incorrect information introduced in conversations.
Authors
Linghao Meng, Feng He, Xuan Yang, Junyuan Mao, Pinze Ren, Deqing Mu, Hesen Yang, Qiankun Li
Abstract
Hallucinated information can propagate through multi-stage LLM systems and become part of the context for subsequent reasoning. Existing studies of post-hallucination reasoning (PHR) mainly characterize changes in final outcomes and aggregate reasoning dynamics, leaving how models resolve hallucinated premises at the response level insufficiently understood. In this work, we introduce PHRBench, a controlled benchmark for behaviorally structured PHR across four domains and 18 large language models. PHRBench characterizes each reasoning trajectory independently of final-answer correctness through Hallucination Compliance, Hallucination Avoidance, and Heuristic Correction, and defines an insightful trajectory as successful correction that ultimately reaches the correct answer. Across 4820 controlled instances, we find that successful recovery remains relatively rare and is associated with more frequent belief updates along the reasoning trajectory. We further find that properties of the hallucinated prompt contain substantial predictive signal for successful recovery, with a lightweight predictor achieving an AUROC of 0.847. These findings provide a behavioral view of post-hallucination reasoning, characterizing how LLMs resolve erroneous context and when successful recovery is likely to occur.