Robot asks better questions to learn tasks with less human effort
Asking Smarter Questions: Targeted Active Preference Learning for Underspecified Rewards
RoboticsArtificial IntelligenceHuman-Computer InteractionMachine Learning
Summary
Teaching a robot what to do can be hard because people often focus on some parts of a task but miss others. The authors found a way for robots to figure out which parts of the task they don’t understand well and then ask smarter, simpler questions about just those parts. They also help people focus on the important details when answering these questions. This method helps robots learn faster and with less effort from people.
What this means in practice
- •For robotics developers: Improve robot learning by targeting questions to unclear task features, reducing teaching time and effort.
- •For interactive ai system designers: Design AI that elicits more useful user feedback by guiding attention with explanations focused on uncertain aspects.
Authors
Helena Merker, Nick Walker, Andreea Bobu
Abstract
Reward learning typically assumes the human's input covers everything that matters for the task. In practice, humans can only attend to so much at once, so their teaching naturally captures some features---or aspects of the task---better than others. Prior work has shown how the robot can identify which features were poorly supervised directly from its teaching data. But the current solution is to ask the human to try again, putting the burden on someone who already struggled to convey those features. We propose shifting this burden to the robot. Our key insight is that the same information that tells the robot what it doesn't know can help it ask better questions. Preference queries offer a low-effort channel for resolving this uncertainty, but only if the robot knows where to direct them and the human knows what to attend to when answering. We act on this insight in two ways: we build an informed prior that targets preference queries at the features that need resolving, and we accompany each query with an explanation that focuses the human's attention on exactly those features. In simulation, the informed prior beats the standard active preference learning baseline by up to $30.8\%$ in the low-query regime, where reducing human effort matters most. In a user study, the informed prior and explanations each significantly improve reward recovery and amplify each other.