Papers for

telepresence system engineers

Papers whose findings have a practical use for this group, as judged from the abstract. Open a paper to read what it means in practice.

Dialogue model generates speech and gestures in real time

Full-Duplex Interaction with Speech, Expressions and Gestures

Abstract: Human face-to-face conversation requires coordinating speech, gestures and expressions across both speaking and listening. We present ELLSA 2, an end-to-end full-duplex dialogue model that jointly optimizes speech interaction and streaming gesture-expression generation. An autoregressive, chunk-wise synthesizer combines pretrained acoustic features with distinct listening and speaking hidden states from the dialogue backbone and speech synthesizer, enabling continuous non-verbal generation even when the model is verbally silent. To translate dialogue understanding into vivid semantic gestures, we introduce contextual semantic calls and template guidance, separating whether to invoke an available semantic gesture from how to realize it. We also establish a curated data and evaluation pipeline for dialog-aware co-speech gesture generation and investigate reference-based multimodal large language models (MLLMs) as automatic judges. Experiments show that ELLSA 2 remains highly competitive with open-source full-duplex speech interaction models and outperforms cascaded baselines across dialog-aware gesture and expression metrics. Semantic template guidance further significantly improves semantic gesture quality. Strong system-level correlations with human judgments support these reference-based scores as useful preliminary metrics for full-duplex and semantic gesture assessment. Together, these results advance dialogue modeling from spoken responses toward coordinated multimodal interaction.

Thu 15 OctArtificial IntelligenceComputer Vision and Pattern RecognitionSound
The gist
Face-to-face conversations involve talking and using gestures and facial expressions together. The authors developed ELLSA 2, a computer model that can talk and generate matching gestures and expressions continuously, even when it is not speaking. Their method uses a special way to decide when and how to produce meaningful gestures based on the conversation. They tested their system against others and found it performs well, especially in making gestures that relate to the meaning of the dialogue. This work helps move computer conversation systems toward more natural and expressive interactions.
Open → 2610.21452v1