SCOPE improves agent learning with selective step self-distillation
SCOPE: Selective Critical-Step On-Policy Self-Distillation for Agentic Reinforcement Learning
Artificial Intelligence
Summary
Teaching AI agents to complete long tasks is tricky because not every step they take is clearly right or wrong. The authors created SCOPE, a way for agents to learn from their own past actions by focusing only on the important mistakes. This selective learning helps the agents improve faster and finish tasks more efficiently. In tests, SCOPE led to higher success rates on tasks involving virtual environments and online shopping.
What this means in practice
- •For software engineers: Improve AI agent training by selectively correcting critical steps in task performance to boost success rates and efficiency in complex virtual environments.
- •For automation system developers: Enhance control policies for automated online services by focusing learning on key decision points, enabling better completion of multi-step workflows.
Authors
Qirui Liu, Wei Cao, Yu Mi, Yichen Sun, Xuekun Zhang, Lei Liu, Yue Shen, Zhixuan Chu
Abstract
Outcome-based reinforcement learning provides coarse supervision for long-horizon LLM agents: failed trajectories may contain correct steps, while successful trajectories can contain errors. Trajectory-level skills can also shift teacher preferences at unrelated steps, introducing off-target distillation signals. We introduce SCOPE, a framework for Selective Critical-Step On-Policy Self-Distillation. After a supervised warm-up, the policy analyzes its own trajectories, generates step-specific hindsight skills, and distills their guidance only at selected error steps. A bounded signed teacher-student coefficient stabilizes auxiliary updates while preserving positive reinforcement and negative correction. A success-relative pairwise length reward further encourages efficient task completion. Experiments on ALFWorld and WebShop demonstrate consistent improvements across multiple model backbones. With Qwen2.5-3B, SCOPE achieves 97.7% success on ALFWorld and 85.2% on WebShop, outperforming SEED by 5.9 and 6.3 percentage points, respectively. On unseen ALFWorld tasks, SCOPE reaches 89.6% success, improving over GRPO by 23.2 percentage points with 46.7% fewer interactions. Scaling to Qwen3-32B, SCOPE achieves 99.2% success on seen ALFWorld tasks and 94.0% on unseen tasks.