Benchmark measures how well AI systems point to objects in conversation
A Benchmark for Spatially Grounded Gesture Generation
Computer Vision and Pattern RecognitionGraphicsHuman-Computer Interaction
Summary
People often use pointing gestures along with spoken language to clearly indicate objects in the environment. The paper presents a new benchmark to test how well AI systems can generate such pointing gestures aligned with speech, using videos from virtual reality conversations where the exact 3D objects being referred to are known. The authors also provide a baseline system and show that while AI can sometimes point more precisely to the correct object than humans, this does not always make the gestures look more natural. This suggests that both accuracy and naturalness must be evaluated separately when studying referential gestures.
What this means in practice
- •For virtual reality developers: Improve interaction realism by adding AI-generated pointing gestures that accurately indicate objects during VR conversations.$Commercial implications: Enables VR platforms to sell more immersive social experiences with reliable gesture-based communication.
- •For human-computer interaction engineers: Develop systems that interpret or generate gestures linked to spoken references in shared environments.
Authors
Anna Deichler, Rishabh Dabral, Fethiye Irmak Dogan, Anindita Ghosh, Jonas Beskow
Abstract
Communication in shared space interweaves verbal and non-verbal signals, and pointing gestures anchor language to the environment: "put the cup on that one" is uninterpretable without the gesture that fixes the referent. Yet no common framework exists for evaluating whether generated gestures indicate their intended referent; distributional metrics reward a gesture aimed at the wrong object as long as it looks natural. We introduce a benchmark for spatially grounded gesture generation, comprising ~2K pointing-annotated clips from naturalistic VR dialogue with ground-truth 3D referents, a task in which systems must decide when, how and where to point within conversational speech, and a protocol that separates temporal alignment, spatial grounding and perceived naturalness. We also provide a flow-matching baseline, MM-Conv-Flow. Evaluating it alongside an independent retrieval-based system and captured human motion, we find that geometric grounding can exceed that of human pointing without any gain in perceived naturalness, showing that referential gesture quality must be measured along separate dimensions.