Reconciling Process Supervision with Outcome-Based Credit in Agentic Policy Optimization
2026-08-31 • Artificial Intelligence
Artificial Intelligence
AI summaryⓘ
The authors explain that traditional reinforcement learning gives the same credit to all decisions in a long task, which is too coarse. They propose TASPO, a method that uses extra information available only during training to more fairly spread credit to specific actions that led to success. TASPO adjusts these credits without changing the overall reward direction, helping the learning algorithm improve and remain stable. Their experiments show TASPO helps agents perform better and handle new tasks more effectively.
reinforcement learningtrajectory advantageself-distillationprivileged informationpolicy optimizationcredit assignmentaction-level supervisionagentic benchmarkslikelihood shifts
Authors
Jingxiao Yang, Wangjie Gan, Yingxuan Zhuang, Wenqi Zhang, Jintao Chen, Xuhong Zhang
Abstract
Outcome-based reinforcement learning provides verified feedback for language-model agents, but assigns trajectory-level advantage uniformly to all decisions, yielding coarse credit over long-horizon interactions. On-policy self-distillation offers finer supervision by re-evaluating sampled behavior with privileged information (PI) available only during training. However, fine-grained supervision is not necessarily fine-grained credit: PI-induced likelihood changes describe how additional information alters policy preference, but do not directly determine how an executable action should inherit the verified task outcome. This creates a supervision-credit gap. Privileged signals may be irrelevant to the current interaction state, operate at a token granularity misaligned with executable decisions, and lack the outcome semantics required for reinforcement. We introduce TASPO, which converts privileged supervision into outcome-grounded action credit. TASPO constructs decision-applicable PI from verified successful experience, aggregates PI-induced likelihood shifts at the executable-action level, and converts relative action support into positive, bounded, mean-preserving weights on the original trajectory advantage. Thus, the verified outcome determines the update direction and average scale, while PI only redistributes credit across actions. Across three agentic benchmarks, TASPO improves over GRPO by 10.6\% and generalizes better to unseen tasks. Further analysis indicates that TASPO reduces supervision mismatch and that action-level assignment stabilizes the policy optimization process. These findings offer the community another interesting perspective.