Reliability-weighted method improves learning from noisy preference data
Reliability-Weighted Direct Preference Optimization: Penalized Weights for Robust Preference Learning
Machine Learning
Summary
When teaching AI to make choices based on preferences, errors and contradictions in the data can confuse the learning process. The authors developed a method that assigns a reliability score to each comparison, helping the system focus more on trustworthy preferences and less on noisy or mistaken ones. Their method finds and reduces the effect of reversed or incorrect comparisons while learning the best policy. Experiments show it improves accuracy and robustness even when a significant portion of the data is faulty.
What this means in practice
- •For language model trainers: Improve fine-tuning of language models by mitigating the impact of mislabeled preference data, maintaining strong performance despite 10-30% flipped labels.
- •For machine learning engineers: Enhance robustness of AI systems learning from human preference data by automatically weighting and correcting unreliable comparisons in training.
Authors
Bin Luo, Param Venkat Vivek Kesireddy, Anil Vallepu
Abstract
Direct Preference Optimization (DPO) treats every reported comparison as correct, yet preference data contain reversed, tied or otherwise noisy comparisons. RW-DPO gives each comparison an unknown reliability weight $w_i\in[0,1]$ that enters the likelihood linearly -- DPO is $w\equiv1$, a reversed comparison $w_i=0$ -- and estimates the weights jointly with the policy under a minimax-concave penalty on $-\log w_i$. The penalty keeps every comparison the model agrees with at exactly $w_i=1$ and acts only on those it contradicts, through closed-form weight updates at negligible cost over DPO. We prove that recovering all flipped comparisons from a single annotation per comparison requires margins of order $\log n$. Furthermore, in a linear-index model, starting from an initial fit pointing in the correct direction, our estimator jointly consistently recovers the preference direction and weights, as well as identifies the flipped comparisons above this margin scale, even when a fixed fraction of comparisons is flipped. In simulations RW-DPO has the lowest average estimation error among DPO, noise-rate corrections and recent robust methods, with far more precise flags than recent label-correction methods. In DPO fine-tuning of a language model with 10-30% flipped labels, its win rate does not fall as the flip rate grows and exceeds that of DPO trained on clean labels, matching the strongest recent method with corrections concentrated on the flipped labels.