LLM agents improve task success by avoiding repeated mistakes
Productive State-Space Exploration for Multi-turn RL in LLM Agents
Artificial IntelligenceMachine Learning
Summary
Large language model (LLM) agents that make decisions over many steps can get stuck repeating the same wrong actions or loops, stopping them from finishing tasks. This happens because they don’t understand the environment well enough and often break its rules. The authors propose a new way to train these agents by giving them a penalty when they repeat states, encouraging them to explore new situations instead. This helps agents learn better behaviors that avoid mistakes and repetitive patterns, improving their overall task performance.
What this means in practice
- •For chatbot developers: Develop chatbots that can complete longer, multi-step conversations without repeating errors or getting stuck in loops.
- •For game ai designers: Create game agents that explore environments more effectively by avoiding repeated invalid moves and reducing cyclical actions.
Authors
Mohammed Haroon Dupty, Yong Liang Goh, Song Guo, Wee Sun Lee
Abstract
Reinforcement Learning (RL) for Large Language Model (LLM) agents in multi-turn, long-horizon environments remains challenging due to inefficient action selection and repetitive behavioral patterns. In particular, agents often become trapped in high-confidence yet unproductive behaviors, repeatedly executing invalid actions or entering action loops that hinder task completion. We observe that these frequently violate environment preconditions, revealing a mismatch between the agent's representation of the internal state and the dynamics of the underlying environment. In addition, repetitive high-confidence actions lead to cyclical behaviors that provide little to no progress towards the goal. We argue that addressing these challenges requires more productive task-aligned exploration to improve the agent's understanding of environment dynamics. Motivated by intrinsic curiosity-driven exploration, we propose a conditional state-space exploration framework that balances task completion with intrinsic information-seeking behavior. Specifically, we introduce an intrinsic reward that penalizes actions whose observation duplicates prior state memory. This provides the benefit of reducing redundant behaviors and encouraging state-space coverage without relying on a general information-theoretic measure. Strong experimental results demonstrate that our approach substantially improves policy learning in multi-turn environments, mitigating invalid actions and repetitive loops.