Agent-G$^2$: Gaussian Guidance for Agentic Reinforcement Learning
2026-08-24 • Artificial Intelligence
Artificial IntelligenceComputation and Language
AI summaryⓘ
The authors study how to help reinforcement learning agents by giving them hints from expert examples to improve learning in tasks that take many steps. Instead of using a fixed amount of hint information, they find it works better to vary this hint length around a range, modeled as a Gaussian distribution, and adjust this based on ongoing experience without extra trial runs. They propose Agent-G², which estimates the best hint length and spread online, considering task difficulty clusters, and shows better results on benchmark tasks with fewer rollouts than previous methods. Their approach avoids costly tuning while adapting hints to different tasks effectively.
reinforcement learningreward sparsityexpert trajectoryguidance depthGaussian distributionpolicy optimizationrolloutshint-based learningtask heterogeneityALFWorld
Authors
Zixuan Wang, Yanrui Miao, Zhengxi Lu, Teng Pan, Yiwen Qiu, Hongxing Li, Peng Qiu, Ruiqing Zhang, Yongliang Shen
Abstract
Hint-based reinforcement learning addresses reward sparsity in long-horizon agentic tasks by retaining a prefix of an expert trajectory before each rollout, letting the policy explore from a state closer to success. Its effectiveness hinges on the guidance depth: how much of the trajectory to keep. Existing methods treat this depth as a deterministic scalar. Scheduled approaches share one value across samples and ignore per-task heterogeneity; per-sample probing estimates it separately at the cost of extra rollouts. We find that useful guidance occupies a band of depths whose informativeness profile is approximately Gaussian around the band center, rather than concentrating at a single optimal point. We propose Agent-G$^2$, a Gaussian guidance framework that draws the depth per task from a Gaussian whose center and spread are estimated online from rollouts already collected for policy optimization, requiring no probe rollouts or learned depth predictor. The center combines a global baseline with per-cluster difficulty, and the spread tracks within-cluster variance. We evaluate Agent-G$^2$ on ALFWorld and WebShop on Qwen2.5-1.5B / 7B-Instruct. Agent-G$^2$ outperforms the strongest hint-based, hint-free, and Aux-RL baselines on ALFWorld by 2.3 / 3.9 / 7.4 points at under one-third the rollout cost of per-sample probing.