Human behaviour models may not improve robot learning outcomes
Behavioural Quality Is Not Learning Utility: Objective Mismatch in Human Behaviour Models for Robot Learning
Robotics
Summary
People often use computer models of human behavior to help train robots, expecting that if the models act like real humans, they'll be good practice partners. This paper shows that models which look like real human behavior don't always help robots learn better policies. The researchers tested this idea in tasks like driving and handing over objects and found that standard ways to measure model quality don’t reliably predict how useful the models are for training robots. Surprisingly, sometimes training robots on different behavior patterns than the actual ones can lead to better learning.
What this means in practice
- •For robotics engineers: Improve robot training by selecting or designing human behaviour models based on downstream learning impact rather than resemblance to real humans.
- •For autonomous vehicle developers: Refine simulation environments by choosing behaviour models that optimize policy learning performance, not just behaviour fidelity, for training control systems.
Authors
Haoyang Jiang, Dana Kulic, Michael Burke, Pamela Carreno-Medrano
Abstract
Human behaviour models are increasingly used to provide simulated human interactions for training robots via reinforcement learning (RL). Existing approaches primarily evaluate these models according to how closely their generated behaviours resemble real human data, while their usefulness as training partners is less commonly assessed. This creates a potential objective mismatch: a model that performs well according to behavioural evaluation metrics may not necessarily provide useful training experiences for downstream robot policy learning. Across two human-robot interaction (HRI) domains, highway driving and object handover, we evaluate different types of human behaviour models using common generative-model evaluation metrics. We consider multiple human trajectory representations and compare the resulting evaluations with the downstream performance of RL policies trained using the corresponding models. We find that existing behavioural evaluation metrics do not consistently predict downstream policy performance across domains and trajectory representations, while their evaluations are sensitive to the choice of trajectory representation. We also show that even when the true testing behaviour distribution is known, training against a different distribution may yield better downstream performance than training directly against the true distribution.