S-SPPO: Semantic-Calibrated Self-Play Preference Optimization
2026-06-01 • Artificial Intelligence
Artificial IntelligenceMachine Learning
AI summaryⓘ
The authors study how to improve large language models so their answers better match what humans prefer. They find that a recent method called SPPO can become unstable when it incorrectly treats very similar answers as big wins. To fix this, the authors create S-SPPO, which uses two new techniques to keep the model's learning balanced and diverse. Their approach helps the model learn more steadily and perform better on tests without needing extra human feedback.
Large Language ModelsDirect Preference OptimizationBradley-Terry modelSelf-Play Preference OptimizationSemantic CalibrationWin-Lose PairsNash EquilibriumLatent SpaceModel AlignmentAlpacaEval
Authors
Xiwen Chen, Wenhui Zhu, Jingjing Wang, Peijie Qiu, Zhipeng Wang, Huayu Li, ZhengXiao He, Xuanzhao Dong, Prayag Tiwari, Mingkun Xu, Yujian Xiong, Feng Luo, Abolfazl Razi, Brendan Hogan Rappazzo, Anderson Schneider, Yuriy Nevmyvaka
Abstract
Aligning Large Language Models (LLMs) with human preferences is often formulated via Direct Preference Optimization (DPO). However, the standard Bradley-Terry instantiation of DPO is limited in modeling common departures from transitivity in human preferences. To address this, recent work has introduced Self-Play Preference Optimization (SPPO), which iteratively refines the policy by training on self-generated win-lose pairs. Our investigation, however, reveals a critical instability in SPPO: the optimization is prone to policy degeneration when the preference oracle assigns overly confident wins to semantically indistinguishable responses. To mitigate this, we propose S-SPPO, a dual-space semantic calibration framework comprising: i) Supervision Calibration via semantic gating, which anneals win rate targets toward the maximum-entropy baseline as semantic overlap increases; and ii) Representation Calibration via latent repulsion to enforce geometric diversity to prevent manifold collapse and maintain latent diversity between chosen and rejected samples. Theoretically, we show that the calibration preserves the constant-sum game structure, facilitating convergence to a Nash Equilibrium. Empirically, S-SPPO avoids the performance degradation seen in prior methods, achieving 52.19% win rate and 47.46% length-controlled win rate on AlpacaEval 2.0 with Llama-3-8B, without using additional human-annotated preferences during training. The code will be available at https://github.com/xiwenc1/s-sppo.