Agentic self-distillation improves language agents using environment feedback
Environmental Feedback Modeling Matters: Rethinking Feedback Treatment in Agentic Hindsight Self-Distillation
Artificial Intelligence
Summary
Reinforcement learning often needs rewards to train language agents, but sometimes these rewards aren’t available. The authors found that simply adding environment feedback to training doesn’t work well enough. They created a new method called SELF that teaches the agent both to understand how the environment reacts to its actions and to improve through self-teaching. This approach helps the agent learn better and work more effectively in interactive tasks.
What this means in practice
- •For language model developers: Improve interactive language agents by integrating environment response models with self-distillation training.
- •For robotics control teams: Enhance agent learning in tasks without explicit rewards by modeling environment feedback jointly with policy updates.
Authors
Hangxi Guo, Fengyuan Liu, Yue Wang, Yuhua Qi, Haoyi Xiong, Fei Sun, Mengnan Du
Abstract
Reinforcement learning is commonly used to train language agents in interactive environments, but cannot be directly applied when rewards are unavailable. Recent methods use environmental feedback as privileged context for hindsight self-distillation, but our analysis suggests that simply conditioning the teacher on feedback is insufficient, motivating us to rethink how environmental feedback is used in agentic self-distillation. Given that environmental feedback contains rich supervision for modeling how the environment responds to agent actions, we introduce \textit{agentic SElf-distilLation with environmental Feedback modeling} (SELF), a framework that jointly optimizes environmental feedback modeling and hindsight self-distillation. SELF learns to predict environmental responses while distilling guidance from a feedback-conditioned self-teacher into the policy. Our analysis reveals a mutually reinforcing mechanism: environmental feedback modeling strengthens hindsight supervision and policy learning, while self-distillation enhances the model's ability to model environmental feedback. With Qwen3-8B, SELF outperforms SDPO and GRPO by 6.4 and 4.1 percentage points in $τ$-bench success rate, and by 10.71 and 3.57 percentage points in AppWorld task goal completion, respectively. These results show that SELF uses environmental feedback more effectively within agentic self-distillation, improving agent capabilities.