Sparse Mixture-of-Experts Reward Models Learn Interpretable and Specialized Experts for Personalized Preference Modeling
2026-06-02 • Machine Learning
Machine LearningArtificial IntelligenceComputation and Language
AI summaryⓘ
The authors focus on improving how AI models learn human preferences to match human values better. They identify that previous methods often assume everyone has the same preferences, which is not true. To fix this, they introduce a model that uses a mix of specialized experts, each focusing on different preference patterns, and encourages the model to route preferences sparsely and distinctly. Their approach helps make the model's decisions easier to understand and better at adapting to individual users without extra labeling. Experiments show this method improves personalized responses and offers insights into how preferences change.
Reinforcement Learning from Human Feedback (RLHF)Preference ModelingSparse Mixture-of-Experts (MoE)Binary Preference DataPersonalizationInterpretabilityReward FunctionExpert DiversitySparse RoutingPost-adaptation
Authors
Yifan Wang, Jinyi Mu, Mayank Jobanputra, Yu Wang, Ji-Ung Lee, Soyoung Oh, Isabel Valera, Vera Demberg
Abstract
Preference modeling plays a central role in reinforcement learning from human feedback (RLHF), enabling large language models (LLMs) to align with human values. However, most existing approaches assume a universal reward function, neglecting the diversity and heterogeneity of human preferences. To address this limitation without additional annotation costs, recent work has proposed learning multiple preference components from binary data and combining them to model individual preferences. Nevertheless, these components often fail to capture coherent and disentangled patterns, limiting their interpretability and effectiveness for personalization. In this work, we propose a sparse Mixture-of-Experts (MoE) reward model that encourages sparse routing and expert diversity during training on binary preference data. Across controlled and real-world experiments, sparse MoE learns interpretable routing patterns and specialized experts. It also improves test-time personalization, and post-adaptation shifts in expert weights provide a qualitative lens for analyzing how the model adapts to personalized preferences.