Act first method speeds up training for multi-turn language agents
Act First, Reason Later: Accelerating On-Policy Distillation for Multi-Turn Agents via Reference-Conditioned Inverse Dynamics
Machine LearningComputation and Language
Summary
Training language agents to interact in multiple turns usually takes a long time because the agent thinks through each step before acting. The authors propose a new way where the agent acts first using hints from a reference, and then reasons later to improve. This method, called ActFirst-OPD, makes training faster without losing quality. Their experiments show that their approach speeds up training by about two to five times across different tasks while still performing as well or better than previous methods.
What this means in practice
- •For virtual assistant developers: Improve training efficiency of conversational AI agents by enabling quicker action responses during learning, reducing development time.
- •For game ai programmers: Accelerate training of agents that perform multi-turn interactions in simulated environments, enabling faster gameplay agent improvements.
Authors
Zubin Zheng, Jiahao Wu, Shaofeng Zhang, Zhirui Zhang, Yew-Soon Ong, Shengcai Liu
Abstract
On-policy distillation (OPD) trains multi-turn language agents with dense teacher supervision on student-generated responses. However, standard think-then-act rollouts require lengthy reasoning before each short action, delaying environment transitions and experience collection. Generating actions directly reduces this delay but can degrade rollout quality. To address this, we propose ActFirst-OPD, an act-first, reason-later training framework that decouples environment interaction from full-response generation. The student infers and executes actions through reference-conditioned inverse dynamics using its current interaction context and a reference next observation, and switches to autonomous next-action prediction when the resulting transition deviates from the reference trajectory. From the collected interaction contexts, the student asynchronously generates full think-then-act responses for token-level teacher supervision. Experiments across 0.6B-, 1.7B-, and 4B-parameter Qwen3 students show that ActFirst-OPD achieves average wall-clock training speedups of $2.3\times$ on ALFWorld, $1.8\times$ on WebShop, and $4.9\times$ on ScienceWorld over Vanilla OPD. It matches or exceeds all compared OPD baselines in mean task success rate across eight of nine benchmark-model settings. These results demonstrate that reasoning need not block acting during multi-turn agent distillation.