Se-DPO: Self-Evolving Token Credit for Direct Preference Optimization
2026-08-10 • Computation and Language
Computation and Language
AI summaryⓘ
The authors identify that in Direct Preference Optimization (DPO), all tokens are treated as equally important for learning preferences, which is not accurate since different tokens contribute differently. They introduce a method called Se-DPO, which adjusts the importance, or 'credit', of each token based on how much it influences the preference outcome, using signals that change as the model trains. This approach does not rely on external tools and adds only a small extra network to help calibrate token importance more precisely. Experiments show that Se-DPO improves performance compared to the original DPO method.
Direct Preference Optimizationtoken creditKL regularizationpreference learningimplicit rewardmodel calibrationtraining dynamicsself-evolving signals
Authors
Wenxiao Zhao, Shu Wang, Ying Nian Wu
Abstract
Direct Preference Optimization (DPO) aggregates token-level log-probability ratios via uniform summation, implicitly treating all tokens as contributing equally to the preference signal. However, the contribution of individual tokens to the preference signal varies. We introduce token credit, which modulates each token's KL regularization based on its contribution to the preference outcome. We derive that effective token credit is proportional to the magnitude of each token's implicit reward, and observe that this quantity evolves substantially during training. This implies that static token credit becomes increasingly misaligned as training progresses. In this work, we propose Se-DPO (Self-Evolving Token Credit for DPO), a live mechanism that derives token credit from the model's own evolving internal signals during DPO training. Since the reward signal varies in reliability across positions, Se-DPO calibrates token credit based on both the strength and the confidence of each token's contribution. Se-DPO requires no external models, adding only a lightweight calibration network with minimal computational overhead. Experiments show that Se-DPO improves over DPO by up to 9.8 points on AlpacaEval~2 and 12.2 points on Arena-Hard.