Selective tie use reduces biases in preference learning models
Selective Tie Learning: Mitigating Spurious Correlations in Preference Optimization
Machine Learning
Summary
Sometimes, computers learn the wrong clues when deciding what people prefer. The authors show that including examples where preferences are tied can help reduce these mistakes. They find smart ways to pick the best ties to add, which works better than picking ties randomly. Their methods also work well for more complex models like large language models.
What this means in practice
- •For machine learning engineers: Improve preference learning models by selectively adding tie data to reduce biases learned from spurious correlations.
- •For natural language processing teams: Enhance large language models to rely less on misleading cues by employing informed tie selection for preference fine-tuning.
Authors
Christian Moya, Purav Matlia, Elliott Thornley, Guang Lin
Abstract
Preference optimization can learn spurious correlations, and augmenting the preference data with ties (equally preferred responses) can reduce spurious reliance. However, ties differ in their corrective effect, making their selection for augmentation important. For log-linear policies, we derive each tie's influence on spurious reliance and show that taking the largest predicted reductions is first-order optimal among subsets of equal size. Computing these influences requires knowledge of the spurious features and the curvature of the training objective, which is often unavailable or expensive to estimate. To this end, we explore using the magnitude of a tie's implicit reward margin and characterize the conditions under which it is a useful proxy for predicted influence. To test the theory and its qualitative transfer to neural models, we estimate influence with a counterfactual proxy for spurious reliance and use a generalized Gauss--Newton approximation to curvature for language models. Across log-linear policies, neural models, and large language models, our selection rules consistently achieve comparable reductions in spurious reliance with fewer ties than uniform selection.