Frontier-OPD improves multi-turn agent learning by targeting key challenges
Frontier-OPD: Discovering the Edge of Agent Capability and Teaching Agents to Go Beyond
Computation and Language
Summary
Training agents to have conversations or complete tasks over multiple steps is tricky because they need both help in difficult spots and the ability to continue on their own afterward. The authors propose Frontier-OPD, a method that finds points where agents struggle but can still improve with a little teacher guidance. By tracking where agents and teachers disagree and checking if the agent can continue successfully, their method picks useful examples to learn from. Tests show this approach helps agents perform better in simulated environments like ALFWorld and WebShop, compared to standard training methods.
What this means in practice
- •For interactive ai developers: Improve multi-turn agent training by targeting key decision points where brief guidance leads to lasting autonomous improvement.
- •For robotics software teams: Enhance robot task learning with focused teacher interventions that identify and improve their current weakest decision steps for better task completion.
Authors
Zheng Jiang, Yiming Chen, Houde Qian, Ling Li, Chaoyang Li, Lifeng Sun
Abstract
On-policy distillation aligns teacher feedback with a student's current behavior, yet acquiring useful supervision for multi-turn agents requires identifying both where guidance is needed and whether the student can sustain the interaction afterward. We introduce Frontier-OPD, which addresses this problem through the student's learning frontier: decision situations that remain challenging but admit useful autonomous continuation after brief teacher guidance. To operationalize this frontier, Frontier-OPD organizes on-policy experience into a shared-state trajectory graph, where trajectory-balanced aggregation of teacher-student disagreement identifies distinct intervention candidates while preserving the histories needed for replay. Short teacher bridges are then assessed by comparing agreement over the bridge and the ensuing student continuation with an equally long segment of the original trajectory, providing a behavioral proxy for guidance that the current student can build on. Accepted teacher responses augment ordinary distillation data, and fresh rollouts renew the frontier estimate as the policy changes. Experiments on ALFWorld, WebShop, and ScienceWorld report higher mean autonomous performance than the compared distillation baselines across the evaluated student scales and teacher configurations, with ablations indicating complementary effects of candidate selection and guidance filtering. In one ALFWorld timing setting, the additional collection cost is offset by a shorter time to a target success rate.