Language model training improved by controlling learning signals
LSC-DPO: Learning-Signal-Controlled Direct Preference Optimization
Artificial Intelligence
Summary
Training large language models to prefer certain outputs can be tricky because the usual method becomes less sensitive as training progresses. The authors studied this problem by focusing on the learning signal that guides updates, finding that adjusting this signal dynamically helps keep training effective. They created a method called LSC-DPO that controls the learning signal to stay in an optimal range. Testing showed this approach improves performance and makes results more consistent regardless of initial parameter settings.
What this means in practice
- •For ai engineers: Implement stable preference-based fine-tuning of language models by dynamically adjusting learning signals for better aligned model behavior.
- •For machine learning platform teams: Integrate adaptive signal control to reduce variation in fine-tuning results across different initial training states, improving model deployment reliability.
Authors
Yang Qu, Yusheng Han, Chengjia Feng, Handan Liu
Abstract
Direct Preference Optimization (DPO) has become a standard reward-model-free approach for aligning language models with preference data. However, as the scaled preference margin grows during training, the logistic DPO loss becomes progressively less sensitive to further changes. We study DPO from a loss-level geometric perspective and identify the sigmoid factor as a learning signal that characterizes the local sensitivity of the objective. Based on this view, we propose Learning-Signal-Controlled Direct Preference Optimization (LSC-DPO), which dynamically regulates the learning signal near a target regime. A log-space analysis establishes conditions for stable tracking of the target learning-signal regime. Experiments on AlpacaEval 2, MT-Bench, and Anthropic-HH show that LSC-DPO consistently improves over DPO and strong preference-optimization baselines. We further find that different coefficient initializations induce distinct transient learning-signal trajectories even when their later signal levels become similar. Based on this observation, we derive a signal-budget compensation rule that adjusts the target learning signal to compensate for these transient differences. The resulting compensation substantially reduces performance variation across coefficient initializations.