MuVAP: Multimodal Multiparty Voice Activity Projection for Turn-taking Prediction in the Wild
2026-06-15 • Sound
SoundArtificial IntelligenceHuman-Computer Interaction
AI summaryⓘ
The authors created MuVAP, a new system that helps predict who will speak next in group conversations using just one microphone and one camera. They made a clever way to track turns between speakers even when many people are talking by simplifying the problem into current and next speaker roles. Since usual conversation datasets are edited and trick the system, they also made a new unedited dataset to train and test their method. Their tests show that MuVAP works better than older methods for predicting speaker turns in conversations with two or three people.
turn-takingmultimodal frameworkvoice activity projectionspeaker trackingaudio-visual datasetmonocular audiosingle-cameranext-speaker predictionRole-Relative Projectionhuman-robot interaction
Authors
Haotian Qi, Gabriel Skantze
Abstract
Current multiparty turn-taking models often rely on complex microphone arrays or multi-camera setups, limiting their applicability in human-robot interaction scenarios. We introduce MuVAP, a causal multimodal framework that extends Voice Activity Projection by grounding acoustic predictions in face tracks, enabling speaker-aware turn-taking predictions from a monaural audio stream and a single camera view. To address the combinatorial complexity of modeling multiple speakers, we propose Role-Relative Projection, which maps any N-speaker interaction onto a fixed current versus next floor-holder state. Because existing audiovisual datasets contain disruptive editing cuts that break causal tracking, we introduce the Audio-Visual Conversation Corpus, a 31-hour dataset of unedited, single-camera multiparty conversations. Evaluations demonstrate that MuVAP outperforms strong baselines on Shift-Hold and next-speaker prediction tasks across two- and three-speaker settings.