Bayesian method learns user preferences from pairwise choices for personalized models
The Missing Coefficients: Bayesian Pairwise Merging for Model Personalization
Machine Learning
Summary
People often find it hard to give exact numerical feedback when personalizing AI models but can more easily choose between two options. This paper presents a method that guesses the hidden importance weights behind these choices using Bayesian inference, turning user decisions into personalized model settings. The method, called Bayesian Pairwise Merging, allows AI models to be adjusted based on just a few comparisons without retraining for each user. It was tested on tasks like medical report summarization, image captioning, and story writing, showing it can capture user preferences effectively.
What this means in practice
- •For healthcare ai teams: Customize radiology summarization models from doctors’ comparative preferences without new training.
- •For content platform developers: Adjust image captioning or story generation models based on user pairwise feedback to better match individual tastes.
- •For customer experience teams: Tailor recommendation systems to user preferences learned from simple choice feedback instead of numeric ratings.$Commercial implications: Enables building personalized recommendation products using lightweight user input, improving user engagement and satisfaction.
Authors
Yaling Shen, Tongtong Wu, Siyuan Yan, Gholamreza Haffari
Abstract
How can we personalize a shared expert library from a user's pairwise choices? Prior work can realize different reward trade-offs by merging reward-specialized experts, given a vector of trade-off weights. In practice, users can more naturally choose between outputs than specify numerical weights. The challenge is therefore to turn these choices into the coefficients required for merging, while accounting for ambiguity when feedback is limited. Our key idea is to treat the unknown reward weights as latent variables: infer a posterior over them from pairwise choices and reward-score differences, and use its mean directly as the merge coefficients. We instantiate this idea as Bayesian Pairwise Merging (BPM), whose posterior also characterizes which reward trade-offs remain plausible given the feedback. We evaluate BPM on radiology summarization, image captioning, and story generation, spanning text-to-text and image-to-text generation. With 100 feedback per simulated persona, BPM achieves macro decided win rates of 91.7%, 77.1%, and 64.3% against uniform merge. For six pairs of simulated personas, each prefers the model fitted to its own feedback, a pattern also observed in a human proof-of-concept. In simulations under BPM's model and prior, its nominal 90% intervals for temperature-scaled reward weights achieve task-averaged marginal coverage of 88.9% and 89.2% with only 10 and 25 comparisons, respectively. BPM thus enables personalization from pairwise feedback without per-user policy training, while characterizing the coefficient ambiguity left by limited feedback.