Inference-time alignment improves model preferences without retraining
Inference-Time Nash Alignment
Artificial Intelligence
Summary
Fine-tuning AI models to follow user preferences usually needs lots of data, computing power, and access to the model itself. The authors studied how to improve models' responses at inference time—when they give answers—without changing their internal settings. They treated the problem like a game between two strategies and created new ways to find a balance that better matches complex preferences. Their methods worked well in tests and got close to the quality of fully fine-tuned models, even keeping stable performance with different settings.
What this means in practice
- •For machine learning engineers: Fine-tune large AI models at inference without needing model parameter access or heavy compute resources.
- •For ai service providers: Improve user-aligned responses from deployed models on demand without retraining or extensive datasets.$Commercial implications: Enables offering customizable AI responses as a service by aligning models post-deployment based on user preferences.
Authors
Hadi Hosseini, Debmalya Mandal, Duohan Zhang
Abstract
Preference-based fine-tuning methods such as RLHF and DPO require substantial compute and large preference datasets. They also need direct access to the model parameters which are not provided by many state-of-the art models. Inference-time alignment offers a cost-effective alternative without updating model parameters. However, existing inference-time methods rely on a scalar reward model derived under a Bradley-Terry assumption, which cannot represent general preferences. Following recent work on fine-tuning with generalized preferences, in this work, we initiate the study of inference-time alignment under general preferences. We formulate the problem as obtaining a Nash equilibrium of a two-player zero-sum game between policies. We propose two algorithms: Best-of-Nash (BoN) and Nash Mirror Descent (NMD). We prove that both algorithms achieve a duality gap that matches the problem lower bound. Empirically, we implement the two methods on three datasets, which shows that our methods substantially outperform the base policy, converging to the performance of the fine-tuned models. Moreover, our results show that NMD remains robust across the regularization parameter.