PolySLGen: Online Multimodal Speaking-Listening Reaction Generation in Polyadic Interaction
2026-04-09 • Computer Vision and Pattern Recognition
Computer Vision and Pattern Recognition
AI summaryⓘ
The authors created PolySLGen, a system that helps AI understand and respond in group conversations like humans do, using speech, body movements, and speaking status. Unlike past methods focusing on one person or just talking, their system considers multiple people interacting with each other, including their nonverbal signals. It uses special parts to combine body poses and social signals from everyone involved to make realistic and fitting responses. Tests showed that PolySLGen works better than earlier systems at making natural, timed reactions in group settings.
multimodal reaction generationpolyadic interactionsnonverbal cuespose fusion modulesocial cue encoderspeaking state predictionbody motion synthesisconversational AIembodied AImotion-speech alignment
Authors
Zhi-Yi Lin, Thomas Markhorst, Jouh Yeong Chew, Xucong Zhang
Abstract
Human-like multimodal reaction generation is essential for natural group interactions between humans and embodied AI. However, existing approaches are limited to single-modality or speaking-only responses in dyadic interactions, making them unsuitable for realistic social scenarios. Many also overlook nonverbal cues and complex dynamics of polyadic interactions, both critical for engagement and conversational coherence. In this work, we present PolySLGen, an online framework for Polyadic multimodal Speaking and Listening reaction Generation. Given past conversation and motion from all participants, PolySLGen generates a future speaking or listening reaction for a target participant, including speech, body motion, and speaking state score. To model group interactions effectively, we propose a pose fusion module and a social cue encoder that jointly aggregate motion and social signals from the group. Extensive experiments, along with quantitative and qualitative evaluations, show that PolySLGen produces contextually appropriate and temporally coherent multi-modal reactions, outperforming several adapted and state-of-the-art baselines in motion quality, motion-speech alignment, speaking state prediction, and human-perceived realism.