Visual cues improve predicting who speaks next in face-to-face talks
Exploring Multimodal Turn-Taking Cues in Face-to-Face Conversation using Voice Activity Projection
Computation and LanguageSound
Summary
People use not just their voices but also their looks and gestures to decide when to take turns speaking in conversations. The authors studied how adding visual signals like where people look or how their faces move can help predict when someone will start talking. They used a special model that guesses future speaking activity and found that seeing facial movements adds important clues. Combining all visual signals worked better than just using sound, but some results changed depending on whether the talks were natural or acted.
What this means in practice
- •For dialogue system developers: Improve automated conversational agents by integrating visual cues to better predict when users will start speaking.
- •For social robotics engineers: Design robots that can more naturally take turns in conversation by using face and body movement signals alongside audio.
Authors
Willem Berner, Julio Cesar Cavalcanti, Kalle Åström, Gabriel Skantze
Abstract
Turn-taking is a fundamental component of spoken interaction, and while humans naturally rely on both verbal and non-verbal signals, dialogue systems usually depend on audio cues alone. This paper investigates whether visual features from face-to-face conversations can enhance turn-taking prediction beyond what is achievable from audio-only. We extend the Voice Activity Projection (VAP) model, a self-supervised transformer-based model for predicting future voice activity, by incorporating visual features extracted from the large-scale Meta Seamless Interaction dataset of dyadic face-to-face conversations. The visual features include gaze direction, head movement, body and hand pose, and facial action units (FAU). For incorporating the visual features, we explore concatenation, cross-attention fusion, delta features, and trainable gating mechanisms. Results show that visual information improves performance over the audio-only baseline, with FAU being significantly more informative than other feature groups. Body and gaze features nevertheless contribute complementary information, as the model combining all features performs best. Furthermore, results indicate that performance on specific tasks varies depending on whether training and test data come from improvised (acted) or naturalistic (non-acted) conversations.