DualTrack synchronizes speech and gestures across multiple languages
DualTrack: Synchronized speech-gesture generation via symmetric coupling of pretrained priors
Artificial Intelligence
Summary
Making computer-generated speech match gestures naturally is hard because pairing speech with motion data is limited. The authors introduce DualTrack, a method that links pretrained speech and motion models to create better synchronized speech and gestures. It works on different languages and represents hand and body motion in detail. Tests show it makes the speech easier to understand and the gestures look more natural than previous methods.
What this means in practice
- •For virtual assistant developers: Generate synchronized speech and gesture animations for multilingual digital avatars to improve user engagement.$Commercial implications: Enables creation of more natural and expressive virtual assistants for global markets by synchronizing voice and gestures in real time.
- •For game developers: Create believable character animations combining speech and hand/body gestures without needing extensive motion capture datasets.
Authors
Yuanzhuo Hu, Zehan Liu, Xiaoyi Qin, Ming Li
Abstract
Joint speech-gesture synthesis must coordinate two modalities despite limited paired data. Existing approaches often lack bidirectional interaction, have limited language coverage, or simplify body and finger representations. We present DualTrack, which couples pretrained speech and motion priors on a shared 12.5 Hz timeline. Causal adapters exchange previous-packet information, while current-state fusion coordinates the streams before they separately complete sixteen-codebook packets. We evaluate 43 BEAT2 recordings in four languages, with speakers held out from joint training and validation. On the shared English/Spanish inputs, without speech or motion prefixes, DualTrack achieves lower word error rate and full-motion Fréchet Gesture Distance, higher beat consistency and speech naturalness than the evaluated GELINA baseline.