Hyperbolic distance improves interpreting inverse reinforcement learning rewards
Hyperbolic Inverse Reinforcement Learning
Machine Learning
Summary
Inverse reinforcement learning (IRL) tries to figure out the hidden goals behind observed behavior by looking at features of those behaviors. The authors introduce a new way to measure rewards using hyperbolic geometry instead of usual straight-line distances, inspired by how some brain circuits organize information. Their method works as well as top techniques and does particularly well when training data is limited or complex. When applied to human navigation in a museum, it revealed more detailed patterns of how people value combinations of exhibit distances, not just each exhibit alone.
What this means in practice
- •For robotics engineers: Improve robots’ ability to learn human preferences in navigation tasks with limited data by modeling rewards using hyperbolic distances between environmental features.
- •For user experience designers: Analyze and predict human movement patterns in complex spaces like museums to inform layout design through interpretable reward structures involving feature conjunctions.
Authors
Iulia Rusu, Anoop Praturu, Sergei Gepshtein, Tatyana Sharpee
Abstract
Inverse reinforcement learning (IRL) seeks to infer the reward function underlying observed behavior, but its predictive accuracy and interpretability depend on how that reward is computed from the underlying features. While prior IRL reward functions have been implemented through a variety of techniques including linear weights, neural networks, or other function approximators, none make assumptions about the geometry of the underlying feature space. Inspired by geometric organization observed in neural circuits, we introduce Hyperbolic inverse reinforcement learning (Hyperbolic IRL), which parameterizes reward as a function of Hyperbolic distance between the feature vector and a learned parameter vector, as well as its Euclidean distance variant. We tested our methods against a range of linear and nonlinear IRL baselines under both deterministic and stochastic policies. Our Hyperbolic approach achieved comparable performance to state-of-the-art models on established benchmarks and excelled in the low data limit and across dense environments. Applied to human navigation among multiple exhibits in a museum room, Hyperbolic IRL reveals reward structure that depends not only on individual exhibit-distance features, but also on non-separable conjunctions between them. Together, these results demonstrate that a Hyperbolic reward parameterization can improve both the predictive accuracy and interpretability of IRL, providing a framework for studying how non-separable, conjunctive structure can begin to explain behavior.