Policy training method improves large language models against errors
Learning Perturbation Robust Policies for LLM Agents with Stable Optimization
Machine LearningArtificial Intelligence
Summary
Large language model agents that learn by trial and error can sometimes become fragile when their decision-making process is disturbed by noise or simplifications. This paper studies how to make these agents more robust during their training so they keep improving even when their internal state is slightly changed or compressed. The authors introduce a new training method called SPrPO that adjusts how the agents are trained based on how sensitive they are to such perturbations. Testing this method on simulated environments showed that it helps the agents remain effective even after different types of internal changes.
What this means in practice
- •For machine learning engineers: Train language model agents that keep performing well even with compressed or noisy internal representations.
- •For automation system developers: Build web and robotic task agents that maintain reliability despite changes in model size or internal computation.
Authors
Pengxin Wang, Yuanzhe LI, Yuxin Ren, Huanrui Yang, Jingdi Chen
Abstract
Reinforcement learning (RL) has become an effective post-training paradigm for long-horizon large language model (LLM) agents. However, we find that the resulting policies can be sensitive to various policy perturbations, such as hidden-state noise, pruning, and quantization. In this work, we study how to improve perturbation robustness during policy optimization. We first introduce the notion of a perturbation robust policy and analyze conditions under which perturbed policy updates preserve stable monotonic improvement. Based on this analysis, we introduce Stable Perturbation-Robust Policy Optimization (SPrPO), which applies adaptive and sensitivity-aware perturbations during RL training. We evaluate SPrPO on ALFWorld and WebShop and conduct systematic experiments across multiple perturbation types and scales, showing improved perturbation robustness while maintaining stable policy optimization.