Risk-Aware Preference Learning for Stochastic Outcomes
Robotics
Summary
The authors studied how robots learn what humans want by observing their preferences, especially when outcomes are uncertain and risky. Typical methods assume people think in simple, average terms (expected utility), but humans often worry more about rare bad things happening and dislike losses more than gains. They tested a model called Cumulative Prospect Theory (CPT), which better matches how people really feel about risk, in robot navigation tasks where avoiding accidents is important. Their results show that using CPT leads to better learning of rewards compared to the usual method. This suggests it's important to consider human risk attitudes when teaching robots from human preferences.
Authors
Yi-Shiuan Tung, Yuni Wu, Wei Jiang, Alessandro Roncone, Bradley Hayes
Abstract
Learning reward functions from human preferences is a widely used approach for aligning robot behavior with user expectations in human-robot interaction. Most existing approaches assume that humans evaluate uncertain outcomes using expected utility (EU), aggregating outcome utilities linearly with their probabilities. However, behavioral evidence shows that humans are systematically risk-sensitive, overweighting rare negative events and exhibiting loss aversion. We study the consequences of this mismatch in social robot navigation, where safety-critical outcomes (e.g., collisions) are rare but highly consequential. We compare EU with Cumulative Prospect Theory (CPT), a nonlinear model of human decision-making, within a Bradley-Terry preference learning framework. Our preliminary experiments show that when preferences are generated by risk-sensitive users, CPT-based learners recover reward functions with substantially lower regret compared to EU-based learners. Our results highlight the importance of modeling human risk sensitivity when learning rewards from preferences over stochastic robot outcomes.