Comparison based method improves preference alignment in large language models
A Zeroth-Order Paradigm for LLM Preference Alignment
Computation and LanguageArtificial IntelligenceMachine Learning
Summary
Aligning large language models with what humans prefer usually involves directly optimizing certain scores, but this can be inefficient or misleading when preferences are subtle. The authors propose a new approach called Comparison-based Preference Optimization (ComPO), which learns from comparisons between outputs without directly optimizing preference scores. They provide mathematical guarantees that ComPO works under certain conditions and show that it improves language model alignment on several popular models. This method also helps reduce issues related to model likelihoods that can skew alignment.
What this means in practice
- •For machine learning engineers: Improve training procedures for LLMs to better reflect human preferences using comparison-based feedback instead of direct likelihood optimization.
- •For ai service providers: Deploy more reliable preference alignment for large language models in customer-facing applications by reducing issues caused by likelihood-based methods.$Commercial implications: Enables creation of alignment-tuned LLM services with more accurate user preference reflection, enhancing user satisfaction and trust.
Authors
Peter Chen, Xi Chen, Wotao Yin, Tianyi Lin
Abstract
Direct preference alignment methods are widely used to align large language models (LLMs) with human preferences because of their computational and memory efficiency. However, likelihood displacement motivates alternative ways to extract information from preference pairs with small likelihood margins. In this paper, we propose and analyze Comparison-based Preference Optimization (ComPO), a zeroth-order alignment method based on comparison oracles. ComPO extracts directional information from these pairs without directly optimizing a differentiable preference loss on them. We establish a convergence guarantee for its basic offline scheme under smoothness, gradient sparsity, and compatibility between the oracle and a latent objective. We further introduce online ComPO, which retains the offline comparison mechanism and uses unlabeled policy generations for reverse-KL control relative to a reference policy. Following the coverage perspective of preference fine-tuning, we establish a performance guarantee for a basic constrained scheme under local coverage and in-distribution pairwise reward accuracy. Experiments on Mistral, Llama, Gemma-2, Qwen3, and Gemma-3 models demonstrate improvements over existing direct alignment methods, including length-controlled win rates, with pair-level diagnostics providing evidence consistent with mitigating likelihood displacement.