PLC-DPO: Posterior Label Correction in Noisy and Ambiguous Preference Optimization
2026-08-31 • Machine Learning
Machine LearningComputation and Language
AI summaryⓘ
The authors identify a problem with an existing method called Direct Preference Optimization (DPO), which assumes all preference data is reliable, but real-world data often isn't. They propose a new method, PLC-DPO, that can detect and adjust for noisy or unclear preferences by deciding if a preference should be trusted, flipped, or treated as a tie. This helps the model learn better from imperfect data. Their experiments across many tests show their method works better than DPO, especially when data contains noise or disagreement.
Direct Preference OptimizationPreference LearningLabel NoisePolicy OptimizationCalibrated MarginPairwise ComparisonsRobust OptimizationNoisy LabelsHuman DisagreementMachine Learning
Authors
Boryeong Cho, Sumyeong Ahn, Se-Young Yun
Abstract
Direct Preference Optimization (DPO) simplifies alignment through pairwise comparisons but assumes all observed preferences are reliable. Real data often violates this assumption, leading to reversed, weak, or ambiguous labels that cause harmful policy updates. To address this, we propose Posterior Label Correction DPO (PLC-DPO) to robustly optimize preferences by routing each pair's training signal as a clean, flip, or tie case. The key idea is to use the calibrated policy-reference margin as online evidence to take appropriate correction actions. This reframes noisy preference learning as actively correcting supervision direction and strength rather than merely filtering suspicious examples. Across 57 dataset-model-benchmark cells, PLC-DPO obtains the best mean win rate against DPO (60.5 vs. 55.5 for the next-best method). Injected-noise and tie stress tests, human disagreement analysis, and self-confirmation diagnostics further show that the routing remains stable and distinguishes flipped from weakly directional pairs.