Feedback enriched environments improve training for long task AI agents
Environments as Scaffold: Enriching Feedback to Bootstrap Self-Evolving Agents in Long-Horizon Tasks
Machine LearningArtificial IntelligenceComputation and Language
Summary
Long, complex tasks are hard for AI agents to learn because they often get little helpful feedback during training. To fix this, the authors suggest changing the training environment to give richer feedback, helping agents learn better and faster. They tested this idea on science and coding tasks and found it made training more stable and encouraged smarter exploration. The feedback helps the AI internalize useful hints, improving performance over time.
What this means in practice
- •For ai engineers: Train autonomous agents on complex tasks more efficiently by embedding enriched feedback into their training environments.
- •For game developers: Create game AI that learns better strategies during training by integrating feedback-enriched environment mechanisms for long-horizon challenges.
Authors
Hongbang Yuan, Zhuoran Jin, Yixin Cao
Abstract
Large Language Models demonstrate remarkable proficiency in static reasoning, yet training them as autonomous agents through Reinforcement Learning (RL) for long-horizon tasks is often hindered by severe reward sparsity. While conventional \textit{agent-side warming} up via supervised fine-tuning (SFT) can alleviate this, it is frequently limited by data scarcity and constrained exploration. To address this, we propose a paradigm shift to \textit{environment-side adaptation} by constructing \textbf{F}eedback-\textbf{E}nriched \textbf{E}nvironments (\textbf{FEEs}). Through a pilot study, we establish a feedback design strategy that reformulates environments by transitioning from action guidance to observation enrichment during the later stages of both intra-episode exploration and inter-episode evolution. Large-scale experiments on SciWorld and BFCL benchmarks using various Qwen3 model scales and RL algorithms such as GRPO, GSPO, and DAPO demonstrate that FEEs consistently yield performance improvements over standard settings. Furthermore, our analysis reveals that training with FEEs \textbf{(1)} stabilizes training dynamics by reducing entropy volatility, \textbf{(2)} facilitates proactive state-space exploration in difficult tasks, \textbf{(3) }ensures the internalization of environmental guidance into policy weights rather than acting as a mere inference-time prior, and \textbf{(4) }identifies intra-group feedback consistency as a critical boundary for stable optimization.