Agentic rl improved by learning from oracle-guided future actions
SERO: Learning from Executable Oracle Continuations for Long-Horizon Agentic Reinforcement Learning
Artificial Intelligence
Summary
When computer programs learn to make decisions over long periods, they usually only learn from what they have tried before. This limits their ability to improve because they never see better actions they haven't tried yet. The authors introduce a method called SERO that lets a special guide (an oracle) take over after the program tries something, showing better future actions. The program then learns from these guided actions, weighted by how well they match what it can do. This helps the program learn more effectively and handle unexpected situations better.
What this means in practice
- •For interactive ai developers: Improve training of agents that interact with complex environments by using oracle-guided future steps to enhance learning efficiency and generalization.
- •For ecommerce automation teams: Enhance web-based shopping assistants to better explore long task sequences by learning from oracle-generated action sequences in training.
Authors
Dingyi Rong, Yue Shi, Chaofan Ma, Zongrui Wang, Jiezhang Cao, Yixin Cao, Liang He, Ning Liu
Abstract
Long-horizon agentic RL is constrained by the futures visited by the current policy. When rollouts fail to reach useful behaviors, improved credit assignment cannot provide supervision for states and actions that were never encountered. On-policy distillation (OPD) introduces teacher guidance on policy-induced states, and recent multi-turn variants improve how this guidance is scheduled, replayed, refined, or validated. However, they do not directly use the oracle-controlled future itself as the training trajectory. We introduce \textbf{State-Rooted Executable Rollout Optimization (SERO)}, which augments on-policy agentic RL with executable oracle continuations. SERO restores a state actually reached by the current policy and lets a training-only oracle continue interacting with the real environment, making the downstream observations and oracle actions themselves available for learning. It then applies \textbf{policy-reachability weighting}, which weights each oracle step by the frozen behavior policy's compatibility with the preceding oracle actions. This allows deeper supervision to become effective as the policy becomes more compatible with the oracle branch. The oracle loss is optimized separately from the base RL estimator and no oracle is used at inference time. We evaluate SERO on two challenging interactive-agent benchmarks, ALFWorld and WebShop, using Qwen2.5-1.5B-Instruct and Qwen2.5-7B-Instruct. On ALFWorld, learning the full executable continuation is substantially more effective than supervising only the root oracle action, while policy-reachability weighting further improves out-of-distribution performance.