Personalized preference learning improves with unlabeled user responses
PerSPL: Personalized Semi-supervised Preference Learning
Machine Learning
Summary
Collecting user-specific preferences to train personalized AI models is often expensive and gives limited data for each person. The authors propose a new method called PerSPL that can learn from lots of unlabeled responses that don’t have user info or preference labels. PerSPL stabilizes the scores across different questions so it can generate usable training signals from unlabeled data. This approach improves accuracy in predicting user preferences in tasks like summarization and chatbot interaction, especially when labeled feedback is scarce.
What this means in practice
- •For product teams for conversational ai: Use unlabeled chatbot responses to improve personalized reward models and make chatbots better at adapting to individual users.
- •For developers of automated summarization tools: Enhance personalized summarization quality by incorporating unlabeled user interaction data to refine models without requiring extensive preference labels.
- •For recommendation system engineers: Incorporate unlabeled user feedback data to boost personalized preference models in recommendation engines, increasing relevance without more labeled data.$Commercial implications: This enables more accurate personalized recommendations for platforms by leveraging unlabeled data, which can be monetized through improved user engagement.
Authors
Xiaoan Xu, Tingzhou Wei, Juncheng Dong, Korel Gundem, Ethan X. Fang, Zhengling Qi
Abstract
Personalized reward models require user-specific preference comparisons, but collecting such feedback is costly and provides only limited supervision for each user. At the same time, large pools of responses can be generated without user identities or preference labels, yet existing personalized reward models leave this data unused. Directly learning from these responses is difficult because pairwise objectives constrain relative scores within a prompt, whereas pseudo-labeling individual responses requires scores that are comparable across prompts for the same user. We propose \textbf{PerSPL} (Personalized Semi-supervised Preference Learning), a backbone-agnostic framework that uses pair-mean score centering to stabilize this cross-prompt score coordinate and iteratively turns unlabeled responses into user-specific training signals. Against seven representative baselines across four benchmarks---PersonalLLM, HelpSteer2, Reddit TLDR, and Chatbot Arena---PerSPL achieves the highest reported mean accuracy among the evaluated methods for both seen and unseen users. The two latter benchmarks provide real-user preference comparisons in summarization and open-domain chatbot interaction, respectively. These results indicate that unlabeled responses can provide additional supervision for personalized reward modeling when per-user preference labels are scarce.