Predicting human interaction with robots is easier for people than AI models
Social Intuition vs. Machine Reasoning: Anticipating Human-Robot Interaction from multiple modalities
Computer Vision and Pattern Recognition
Summary
Humans find it natural to guess when someone wants to interact with a robot using both body posture and video cues. The authors studied how well people and different AI models can predict this intention from service robots' viewpoints. They discovered that humans do better than pose-based AI models but only slightly. When given full video views, humans outperform advanced vision-language AI models by a bigger margin. The study suggests that current AI struggles to match human social intuition despite having strong reasoning abilities.
What this means in practice
- •For robotics engineers: Improve robot systems by incorporating social intuition cues humans use to better predict when people want to interact.
- •For computer vision developers: Design vision-language AI models that combine reasoning with better social context understanding for interaction prediction.
Tested on one dataset.
Authors
Raphael Lorenzo-Louis, Bertrand Luvison, Serena Ivaldi
Abstract
Anticipating whether a person will interact from one's own perspective is a highly intuitive task for humans, that relies on a combination of cues. We investigate how humans perform at predicting a person's intention to interact from a service robot's point of view, using pose-only or full video input, then benchmark different lightweight pose-based models and state-of-the-art vision-language models. We conducted our benchmark on the HUI360 dataset on a fixed pilot subset of 100 test tracks (25 positive, 75 negative). We found that with pose-only input, human annotators outperform lightweight trained pose models but not by large margins (+0.08 in F1-Score). But when given full egocentric video with a target bounding box, human annotators perform substantially better and largely outperform the Vision-Language Models (+0.2 in F1-Score). We also compared VLMs of different size and under different input conditions, and found that the best results do not correlate with model size. Our result confirms that predicting interactions is a challenging task for social robots and that reasoning-capable models are necessary but their actual reasoning capabilities alone do not suffice to match the social intuition of humans.