Large language model agents fix false claims about entities
Phantom Merge: When Your Large Language Model Agents Pick One but Tell You About Another
Artificial IntelligenceComputation and Language
Summary
When language AI systems talk about different things at once, they sometimes mix up which facts belong to which thing. The authors call this mistake Phantom Merge, where an AI says something about one item but really means another. They studied how often this happens and why, then made a two-step method to spot and fix these errors by checking the AI's internal signals and evidence. Their approach greatly reduced the mix-ups in tests with shopping examples.
What this means in practice
- •For software developers: Improve AI assistants that handle multiple items by detecting and correcting false claims tied to wrong entities in user queries.
- •For customer service teams: Reduce errors in AI-generated responses about products or orders by verifying claims relate to the correct item before delivering answers.
Authors
Wangshu Zhu, Qianwen Guo, Yushun Dong
Abstract
LLM agents increasingly operate over multiple entities, where reliable response generation requires not only retrieving relevant evidence but also preserving its association with the correct entity. We identify Phantom Merge (PM), an entity-evidence binding failure in which an agent attributes unsupported claims to its committed entity, potentially even when the task is completed correctly. We formalize PM at the claim level and characterize its prevalence and error subtypes across four agent domains and four backbone families. To address this problem, we propose a two-stage framework for claim-level risk estimation and correction. First, Anchor-Grounded Risk (AGR) combines a representational readout of the agent's pre-emission hidden state with Jacobian-based functional information, providing complementary slot-level and anchor-value risk scores. Second, Fixed-Anchor Correction (FAC) audits flagged claims and selectively revises them using observed anchor evidence or removes them when no supported replacement is available. On the held-out Shopping detection cohort, AGR-slot achieves an AUROC of 96.2% and an average precision of 97.3%, outperforming the evaluated detection baselines. In the end-to-end Shopping mitigation evaluation, FAC reduces trajectory-level PM from 51.4% to 6.2%, with an output-to-original claim ratio of 80.7%. These results demonstrate the prevalence of entity-specific grounding failures and the effectiveness of combining internal risk estimation with evidence-constrained claim correction. Our code is publicly available at https://github.com/LabRAI/PhantomMerge.