Agentic reinforcement learning improved by subgoal plans and graph credit assignment
SGR-OPSD: Learning from Subgoal Plans with Graph-Regularized Credit Assignment for Agentic Reinforcement Learning
Artificial Intelligence
Summary
Teaching AI agents to complete complex tasks is hard because it's difficult to know which actions really help progress toward the goal. The authors propose a method that breaks tasks into smaller subgoals generated by large language models and uses a graph-based system to better assign credit to important actions. This combination helps agents learn more effectively by giving clearer signals about which steps matter. Their approach improves success rates on several benchmark tests compared to existing methods.
What this means in practice
- •For autonomous robot developers: Guide robot actions in multi-step tasks by using AI-generated subgoal plans combined with graph-regularized credit assignment for improved learning efficiency.
- •For software engineers building virtual assistants: Enhance virtual assistant decision-making by integrating subgoal-based supervision and calibrated credit assignment to improve task completion accuracy in complex environments.
Authors
Shu Liu, Yang Chong, WenShi Chen, Man Lan
Abstract
OPSD provides dense supervision for agentic reinforcement learning through privileged information feedback. However, existing privileged information often lacks explicit task-progression planning, while teacher-student log-probability gaps reflect shifts in model preference rather than true task progress. To address these limitations, we propose Subgoal-Guided Graph-Regularized OPSD (SGR-OPSD), a framework that combines subgoal supervision with interaction-derived evidence for turn-level credit assignment. Specifically, SGR-OPSD uses LLM-generated subgoal plans to provide dense action-level guidance, converting them into advantage weights via a constrained, KL-regularized objective. To ground credit in observed outcomes, we construct a Subgoal Progress Graph that estimates empirical returns from sampled transition frequencies. The graph compares empirical action returns against the state average to derive preferences that calibrate the teacher's credit weights. Evaluated on ALFWorld, WebShop, and Search-QA across two model scales (Qwen2.5-3B and 7B), SGR-OPSD consistently outperforms GRPO and strong self-distillation baselines, improving success rates over GRPO by up to 7.4 percentage points and reaching 85.8% on ALFWorld with Qwen2.5-7B. Ablation studies confirm that subgoal guidance and graph-based weight calibration are key to these performance gains.