Direct preference optimization produces sparse updates in language model training
On the Sparsity of Direct Preference Optimization: Weight Disentanglement in the NTK Regime
Machine Learning
Summary
Training language models to follow human preferences often changes many model settings, but some methods surprisingly update only a few parts at a time. The authors studied Direct Preference Optimization (DPO) and found that it naturally focuses updates on a small set of important examples and parameters. This helps keep model changes more organized and reduces conflicts when training on multiple tasks. Their work explains why DPO behaves this way and shows it can help models balance different skills better.
What this means in practice
- •For machine learning engineers: Optimize language models using DPO to make parameter updates sparse and reduce interference between tasks during fine-tuning.
- •For ai platform developers: Design model merging systems that exploit weight disentanglement properties proven for DPO-trained models to combine models effectively.
Authors
Kensuke Sasaki, Issei Sato
Abstract
Direct Preference Optimization (DPO) has been widely adopted for aligning language models with human preferences. Recent empirical work has revealed that DPO induces remarkably sparse parameter updates, yet no theoretical explanation for this phenomenon has been offered. We analyze DPO's optimization dynamics under a neural tangent kernel linearization and show that the logistic structure of the DPO loss gives rise to an implicit maximum-margin bias. Building on this observation, we introduce the concept of dual sparsity: the DPO update direction is a sparse linear combination (data-level sparsity, driven by support vectors) of sparse gradient feature vectors (parameter-level sparsity, arising from gradient cancellation between preferred and dispreferred responses). We formalize this structure in our main theorem, bound the number of support pairs by a margin-dependent spectral quantity, and present a framework relating dual sparsity to weight disentanglement---a condition favorable for model merging. Experiments on GPT-2 Small and Medium and on 7--9B instruction-tuned models (Llama-3-8B, Mistral-7B, Gemma-2-9B) with the Anthropic HH-RLHF and UltraFeedback datasets confirm both sources of sparsity, and experiments on GPT-2 show that DPO reduces cross-task interference by up to $35{\times}$ compared to supervised fine-tuning.