Multiscale policy optimization improves long task performance in language agents
Multiscale Policy Optimization for Long-Horizon LLM Agents
Artificial Intelligence
Summary
When AI language agents work on tasks that take many steps, it's tricky to know which parts of their responses are helping or hurting. The authors noticed that the value scores used to guide training change mostly between big chunks of work, called turns, rather than between individual words. They created a new training method that looks at both the whole turn and the words inside it to better teach the agent. This method was tested on various tasks and showed steady improvement over previous approaches.
What this means in practice
- •For ai application developers: Train language agents to perform complex multi-step tasks more effectively by improving credit assignment during training.
- •For automation engineers: Enhance tool-using AI agents to make better decisions across extended interactions by optimizing training at multiple response scales.
Authors
Tao Ren, Hui Yang, Jia Deng, Tingyang Chen, Qiyao Wang, Shuo Tang, Ziyang Zeng, Yuhua Jiang, Guanghao Li, Huatong Song, Yining Li, Yifan Luo, Kunbao Ding, Feng Chang, Chuan Hao, Ran Tao, Bryan Dai, Jiafeng Liang, Jinyang Jiang
Abstract
Long-horizon LLM agents interact with their environments through complete assistant turns while generating each action token by token. We observe that critic values vary primarily across turns, motivating optimization anchored at turns with finer credit allocation among tokens. We propose Multiscale Policy Optimization (MPO), combining Multiscale Advantage Estimation (MAE) and turn-level importance sampling to align credit assignment and policy updates at the turn scale. Extensive experiments across search, tool-use, and automation tasks with 4B and 30B-A3B models demonstrate consistent gains over strong RL baselines.