Large language model agents avoid tool misuse with new inference method
Understanding and Mitigating Hallucination Escape in Tool-Using LLM Agents
Computation and LanguageArtificial Intelligence
Summary
Large language models (LLMs) sometimes pick the wrong tools or make errors when using external tools because they imagine things that aren't true, known as hallucination. The authors found that previous fixes only reduce this problem for specific tool setups but make it worse for others—a failure they call Hallucination Escape. They discovered this happens when the model's natural preferences clash with the tool setup, and past fixes actually strengthen these biases. To fix this, they created EscapeGuard, a method that works during inference without retraining, helping LLMs choose tools correctly across different setups and greatly reducing hallucination errors.
What this means in practice
- •For software engineers building ai assistants: Enable AI assistant agents to reliably select and use external APIs without hallucinating incorrect tool calls across various configurations.
- •For enterprise ai platform teams: Reduce errors in tool-using AI agents interacting with diverse enterprise systems by applying a training-free method that adapts during inference.
Authors
Peigui Qi, Kunsheng Tang, Yide Song, Weiming Zhang, Nenghai Yu
Abstract
Large language models (LLMs) increasingly serve as autonomous agents that invoke external tools. However, this capability introduces tool hallucination, selecting incorrect tools or generating invalid calls. Existing mitigation methods report substantial improvements, yet we identify a previously overlooked failure mode that we term Hallucination Escape. These methods reduce hallucination on the tool configuration they are tuned on but increase it on other configurations, canceling out the gain. We further investigate this phenomenon and find that hallucination rises sharply when a model's intrinsic tool-use tendencies conflict with the current tool configuration, and that existing methods reinforce rather than suppress these tendencies, which in turn contributes to hallucination escape. Building on these findings, we propose EscapeGuard, a training-free inference-time method that combines conflict-aware gating with configuration-derived attention enhancement to mitigate tool hallucination while preventing hallucination escape. Across six benchmarks on various models, EscapeGuard reduces tool-selection hallucination by 9.0 pp and suppresses hallucination escape, lowering the cross-configuration mean by 23.7 pp and achieving an 89.1% net improvement in paired-query evaluation. We hope this work can encourage evaluation beyond a single tool configuration and pave the way for more reliable tool-using LLM agents.