LLM agents learn to misuse tools due to misleading cues
Spurious Tool Use: When RL Agents Learn the Wrong Reason to Act
Computation and Language
Summary
Sometimes AI agents trained to use tools like web search get tricked into using these tools for the wrong reasons, just because certain hints appear in their instructions. The authors showed this happens when the AI learns to use a tool well but then starts relying on these misleading hints instead of actual need. They tested this with fake question-answer tasks and showed that when the hints and the tools match up in meaning, the problem gets worse. They also demonstrated a way to fix this by rewarding the AI for only using tools when really necessary, which helps stop the bad habit.
What this means in practice
- •For ai product developers: Improve AI assistants by reducing unnecessary tool calls, making their responses more efficient and relevant.
- •For software engineers: Use decision-level reward techniques to train more robust AI agents for complex multi-tool systems.
Authors
Yiwei Yang, Haoxiang Zhang, Bingbing Wen, Yao Lu, Yuchen Wu, Lei Zhang, Julian McAuley, Pan Lu, Bill Howe
Abstract
Large language model (LLM) agents increasingly interleave natural language reasoning with external tools such as web search and code execution. These tool-use policies are often optimized via reinforcement learning (RL), which can amplify spurious correlations in the training data. In this work, we study when and why RL-trained agents learn shortcut tool-selection policies: invoking tools based on superficial prompt cues rather than genuine task requirements. We construct controlled synthetic environments combining factual question answering and mathematical reasoning tasks, and inject cues that are strongly correlated with specific tools during training but causally irrelevant to tool necessity. Across counterfactual evaluations where cues are present but the associated tools are not required, agents exhibit substantial shortcut behavior, with spurious tool invocation rates increasing by up to 39 percent. However, shortcut formation is not universal: across the conditions we test, it arises only when the agent has already learned to use the target tool reliably, suggesting that task competence, rather than dataset imbalance alone, is a key factor in shortcut learning. A swapped-cue analysis further shows that semantic alignment between cues and tools substantially amplifies this effect. To mitigate these failures, we introduce a dense, decision-level reward in which an LLM judge evaluates the necessity of each tool call. This tool-necessity reward effectively suppresses cue-driven tool use while preserving task performance, providing a practical approach to improving the robustness of LLM agent tool-use policies.