Method reduces expert data for learning transferable rewards
Inferring Transferable Rewards via Active Inverse Reinforcement Learning
Machine LearningHuman-Computer Interaction
Summary
Getting expert examples to teach machines how to learn rewards that work in many different situations can be very expensive. The authors propose a method that first tries different scenarios to get a good guess of the reward, then actively picks scenarios to better understand and reduce uncertainty about the expert's true reward. This approach reduces how many expert examples are needed and works regardless of how many scenarios are available. The result is a more efficient way to learn rewards that transfer across different settings.
What this means in practice
- •For robotics engineers: Teach robots shared goals efficiently across environments with different physics using fewer expert demonstrations.
- •For autonomous vehicle developers: Improve learning of driving policies that adapt to varying road and traffic conditions with limited expert driving data.
Authors
Victor Villin, Till Freihaut, Andreas Schlaginhaufen, Christos Dimitrakakis, Maryam Kamgarpour, Giorgia Ramponi
Abstract
Inverse Reinforcement Learning (IRL) aims to recover expert rewards that transfer across environments with differing dynamics. Prior work has shown that such transfer is possible when gathering expert demonstrations across sufficiently different dynamics. In practice, however, expert demonstrations are costly, making it critical to select the right dynamics to reduce the quantity of expert data required for learning transferable rewards. We propose a two-phase approach: we first cycle through the available dynamics to obtain a reliable reward estimate, then actively select dynamics to reduce our uncertainty about the expert reward. Crucially, this yields novel sample complexity guarantees on transfer that hold for any dynamics, with a rate independent of the number of available dynamics. Empirically, our method significantly reduces the number of expert demonstrations needed to learn a transferable reward.