Tool-calling reinforcement learning improves accuracy and reduces errors
SLCA-GRPO: Resolving Cross-Segment Credit Misattribution in Tool-Calling RL
Artificial IntelligenceMachine Learning
Summary
When AI systems use tools and explain their actions in words, it can be hard to teach them properly because rewards get mixed up between tool use and explanations. The authors created a new method called SLCA-GRPO that carefully separates rewards for tools and language parts, so the AI learns better. They also built a simulator to train and test this method efficiently. This approach helps AI models get more accurate, use fewer tools unnecessarily, and learn faster than older methods.
What this means in practice
- •For software developers: Improve AI assistants that call external tools by reducing errors from mixing tool usage and language output during training.
- •For cloud service engineers: Enhance automated API testing environments by using simulated tool-calling LLMs for scalable and stable training of RL agents.
Authors
Yan Zhan, Shaobo Liu, Qiunan Liu, Yuanjun Shi, Siqi Xu, WeiYi Hou, Xiang Xu, Zekang Li, Weizhou Pan, Jiahong Yan
Abstract
Tool-calling agents produce heterogeneous outputs, interleaving structured tool invocations with user-facing natural language summaries. This output heterogeneity presents a structural failure mode in standard on-policy Reinforcement Learning (RL): algorithms like GRPO indiscriminately broadcast a homogeneous trajectory-level scalar advantage to all tokens. Consequently, gradient noise from summary generation leaks into tool-decision tokens, causing cross-segment credit misattribution and brittle optimization. In this work, we propose SLCA-GRPO, a framework incorporating Segment-Locked Credit Assignment (SLCA). To enable scalable exploration without costly real APIs and stable training, we first construct the Schema-Guided LLM Simulator (SGLS) as foundational training infrastructure. Building on this, SLCA decouples advantage estimation at the structural segment level within a single group of rollouts, without requiring additional rollouts from intermediate states. Supported by Hierarchical Rewards (HierR), SLCA routes execution advantages to tool tokens and preference advantages to summary tokens, eliminating advantage contamination (the dominant cross-segment credit misattribution channel) within each policy update. On a 7B backbone, SLCA-GRPO accelerates convergence and outperforms standard GRPO, ToolPO, and RLTR by +2.53 pp on in-domain evaluation, +1.36 pp on the Berkeley Function-Calling Leaderboard (BFCL), and +9.15 pp on $τ^2$-Bench under the same training budgets, achieving higher accuracy with reduced tool redundancy and costs.