ACPO: Adaptive Credit Policy Optimization via Fine-Grained Surrogate Entropy
2026-07-03 • Machine Learning
Machine LearningArtificial Intelligence
AI summaryⓘ
The authors study a way to help large language models learn better from reinforcement learning, especially when rewards are rare and hard to assign to individual words. They propose a new method called Adaptive Credit Policy Optimization (ACPO) that focuses more on uncertain or overconfident word choices depending on whether the outcome was good or bad. This method aims to give better guidance during training by carefully adjusting how each word is credited. Their tests on math and coding challenges show ACPO improves performance compared to other strong methods.
Reinforcement LearningLarge Language ModelsCredit AssignmentPolicy OptimizationEntropySurrogate EntropyToken-level RewardsMathematical ReasoningCoding BenchmarksPolicy Gradient
Authors
Zijun Xie, Yuyang You, Yongzhi Li, Enlei Gong, Zeyu Chen, Quan Chen, Yanhua Cheng, Peng Jiang, Yadong Mu
Abstract
Reinforcement Learning (RL) has substantially improved the reasoning ability of large language models (LLMs), but sparse outcome rewards still make token-level credit assignment difficult. Existing scalable RL methods typically assign trajectory-level rewards uniformly across tokens, while recent entropy-aware approaches either rely on coarse detached heuristics or directly optimize true entropy, which can introduce non-local gradient components misaligned with sampled-token policy updates. We propose Adaptive Credit Policy Optimization (ACPO), a token-level credit assignment framework based on a mode-local surrogate entropy. ACPO asymmetrically modulates policy updates by emphasizing uncertain decisions in successful rollouts and overconfident tokens in failed rollouts. We show that the surrogate admits deterministic entropy bounds and, under modal alignment and proximal updates, preserves the policy-gradient direction to leading order. Experiments on mathematical reasoning and coding benchmarks, including AIME 2025 and HumanEvalPro, show that ACPO consistently improves over strong RL baselines such as DAPO, GTPO, and SAPO.