Multimodal voice activity predicts turn-taking for social robot mediation
Multimodal Voice Activity Projection for Social Robot Mediation: Expected Behavior and Deployment Constraints
Robotics
Summary
Knowing when to speak or stay quiet is important for robots that help people talk to each other. The authors developed a system that uses sound and video to guess who will speak next and whether someone will keep talking or let another person speak. This helps the robot behave politely by waiting, listening, or gently joining the conversation. They tested their method on several datasets and discussed how to make it work in real time with good input quality.
What this means in practice
- •For social robot developers: Enable social robots to predict conversation flow and decide when to speak or listen during human interactions.
- •For human computer interaction engineers: Improve interactive systems by adding future voice activity projection to anticipate user turn-taking and overlap in conversations.
Authors
Antonio Cano, Guillermo Perez, Luis Merino, Randy Gomez
Abstract
Turn-taking prediction is especially relevant for social robots that act as mediators in human-human interaction, where the expected action is often not to speak, but to orient, wait, avoid interruption, or prepare a balanced intervention. This paper presents Multimodal Voice Activity Projection (MM-VAP) as a human state-aware perception layer for future robot mediation behavior. The model estimates the future evolution of the conversational floor from synchronized audio-visual evidence and derives turn-taking events such as Hold, Shift, Shift prediction, Backchannel prediction, and overlap-related states. The approach uses VA-related pretrained audio-visual encoders, LoRA adaptation, inter-speaker attention, and zero-shot event inference from future voice activity projections. Experiments on NoXi, NoXi+J, and Haru EDR support the feasibility of this formulation, especially for floor management events that can be connected to gaze preparation, active listening, and conservative intervention. Finally, the paper defines the expected robot output interface and discusses the main deployment constraints, including real-time inference, preprocessing latency, multimodal synchronization, and input-quality monitoring.