Environment agent and user co-evolution improves multi-turn interactions
EAU: Environment-Agent-User Co-evolution for Training User-Centric Agents
Artificial IntelligenceComputation and LanguageMachine Learning
Summary
Training AI agents to chat and complete tasks over many turns is tricky because the experience depends on the agent, the user it talks to, and the environment where they interact. The authors created a system called EAU that watches how the agent, user, and environment work together during training. When problems happen, EAU changes parts of the user or environment setup to help the agent learn better. This approach leads to stronger agents that finish more tasks successfully across several tests.
What this means in practice
- •For chatbot developers: Improve chatbot training by dynamically adjusting user simulations and environments based on agent behavior to enhance multi-turn task completion.
- •For automated customer support teams: Create more reliable support agents that adapt training setups automatically to handle real-world user variability and tool inconsistencies.
Authors
Min Cai, Dan Zhang, Gao Xing, Libin Ma, Zhaohui Xu, Xi Ye
Abstract
Training LLM agents to interact with users over multiple turns is challenging because the learning signal depends not only on the agent, but also on the user simulator and environment, which jointly shape the agent's experience. Simulated users can be stochastic or inconsistent, while environments may contain unreliable tools, incomplete state coverage, and incorrect verifiers. Prior work mainly addresses the resulting instability through reward shaping or denser supervision, while treating these components as fixed. We instead introduce EAU, a meta-agent framework for environment--agent--user co-evolution. During training, EAU monitors trajectories, tool traces, rewards, and training statistics; diagnoses failures and uninformative interactions; and proposes targeted modifications to the user simulator, environment, reward design, and RL configuration. Validated changes are hot-swapped into the ongoing run, allowing the training setup to evolve together with the policy. These adaptations also persist across runs, so later training can build on previously discovered improvements rather than restarting from a fixed setup. We enable this process through an editable symbolic representation of user and environment dynamics, while the agent remains a neural LLM optimized with RL. Across three interactive-agent benchmarks and two backbone models, EAU consistently outperforms prior RL methods, improving the overall score over the strongest baseline by 14.7--15.0% and increasing successful task completion on ColBench by up to 36.7%, showing that co-evolving the training system with the policy can substantially improve multi-turn RL.