UBP2: Uncertainty-Balanced Preference Planning for Efficient Preference-based Reinforcement Learning
2026-06-17 • Machine Learning
Machine LearningArtificial IntelligenceRobotics
AI summaryⓘ
The authors present a new way to teach computers to learn what we want by comparing different behaviors, instead of giving explicit rewards. Their method, called Uncertainty-Balanced Preference Planning (UBP2), actively chooses actions by considering how uncertain it is about the reward, environment, and future outcomes, balancing between trying what seems best and exploring to learn more. This approach avoids random guessing and improves learning efficiency, especially early on. Tests on the Meta-World tasks show it learns faster than older methods. The authors also provide theoretical guarantees that their method performs well over time.
Preference-based Reinforcement LearningReward ModelModel-based RLExploration vs ExploitationEpistemic UncertaintyEnsemblesMeta-World BenchmarkRegret GuaranteesFinite-horizonInfinite-horizon
Authors
Mohamed Nabail, Leo Cheng, Jingmin Wang, Nicholas Rhinehart
Abstract
Preference-based RL provides an approach to learning reward models from pairwise comparisons of behaviors, bypassing the need for explicit reward design. However, existing methods typically rely on passive data collection and suffer from poor sample efficiency, especially during the early stages of learning. We introduce a model-based approach that actively directs exploration by jointly reasoning over uncertainties in the reward, dynamics, and value functions. Our method, Uncertainty-Balanced Preference Planning (UBP2), uses ensembles of reward, dynamics, and value function models to evaluate candidate trajectories according to a unified score that combines expected reward, terminal value, and epistemic uncertainty. Planning under this objective yields an explicit tradeoff between exploitation and information acquisition without requiring ad hoc exploration heuristics. Under standard regularity assumptions, we establish sublinear regret guarantees for both finite-horizon and infinite-horizon settings. Empirically, experiments on the Meta-World benchmark show UBP2 achieves substantially higher sample efficiency than model-free preference-based methods and non-optimistic model-based baselines.