Dialogue model generates speech and gestures in real time
Full-Duplex Interaction with Speech, Expressions and Gestures
Artificial IntelligenceComputer Vision and Pattern RecognitionSound
Summary
Face-to-face conversations involve talking and using gestures and facial expressions together. The authors developed ELLSA 2, a computer model that can talk and generate matching gestures and expressions continuously, even when it is not speaking. Their method uses a special way to decide when and how to produce meaningful gestures based on the conversation. They tested their system against others and found it performs well, especially in making gestures that relate to the meaning of the dialogue. This work helps move computer conversation systems toward more natural and expressive interactions.
What this means in practice
- •For conversational ai developers: Create virtual agents that speak and naturally accompany speech with coordinated gestures and facial expressions in real time.$Commercial implications: Enables richer, more engaging virtual assistants and customer service bots that behave like humans, improving user experience and commercial appeal.
- •For telepresence system engineers: Integrate synchronized speech and gesture generation to make remote communication avatars more lifelike and expressive.
Authors
Siyin Wang, Wenyi Yu, Terumi Chiba, Xianzhao Chen, Xiaohai Tian, Jun Zhang, Lu Lu, Chao Zhang
Abstract
Human face-to-face conversation requires coordinating speech, gestures and expressions across both speaking and listening. We present ELLSA 2, an end-to-end full-duplex dialogue model that jointly optimizes speech interaction and streaming gesture-expression generation. An autoregressive, chunk-wise synthesizer combines pretrained acoustic features with distinct listening and speaking hidden states from the dialogue backbone and speech synthesizer, enabling continuous non-verbal generation even when the model is verbally silent. To translate dialogue understanding into vivid semantic gestures, we introduce contextual semantic calls and template guidance, separating whether to invoke an available semantic gesture from how to realize it. We also establish a curated data and evaluation pipeline for dialog-aware co-speech gesture generation and investigate reference-based multimodal large language models (MLLMs) as automatic judges. Experiments show that ELLSA 2 remains highly competitive with open-source full-duplex speech interaction models and outperforms cascaded baselines across dialog-aware gesture and expression metrics. Semantic template guidance further significantly improves semantic gesture quality. Strong system-level correlations with human judgments support these reference-based scores as useful preliminary metrics for full-duplex and semantic gesture assessment. Together, these results advance dialogue modeling from spoken responses toward coordinated multimodal interaction.