Streaming human motion reacts in real time to spatial audio cues
Sonic-AMG: Streaming Human Motion Generation from Spatial Audio
Computer Vision and Pattern RecognitionGraphicsMachine LearningRoboticsSound
Summary
People naturally move when they hear sounds and can tell where they come from, but previous computer models needed the whole sound in advance to create matching movements. The authors created Sonic-AMG, a system that listens to sounds as they happen and makes people-like motions instantly, using a new way to understand both the sound and its location. They also figured out how to train the system so it works well when used live, even though it only gets partial information at a time. Tests show Sonic-AMG makes more realistic and timely reactions than earlier methods.
What this means in practice
- •For virtual reality developers: Generate real-time avatar motions that respond naturally to nearby spatial sounds during live user interactions.$Commercial implications: Enables VR products to offer more immersive human-like responses to audio events, enhancing user experience.
- •For video game studios: Create characters that react instantly and realistically to in-game sound events with appropriate body movement.
Authors
Jingcheng Ni, Zekun Li, Zhiyang Dou, Xiaoyan Cong, Ziyao Zeng, Hongyang Du, Randall Balestriero, Taku Komura, Srinath Sridhar
Abstract
Modeling human reactions to spatial audio requires understanding both what is heard and where it comes from. Prior work generates motion offline from complete acoustic and spatial input sequences. Humans, however, react to sounds without knowing what they will hear next. We present Sonic-AMG, to our knowledge the first framework for streaming human motion generation from spatial audio. The model encodes acoustic content and body-relative source geometry as separate tokens. Conditioned on these tokens, a causal transformer with a flow head generates continuous motion latents, from which a causal decoder reconstructs motion segments. Standard teacher-forced training derives motion histories and body-relative source locations from ground-truth motion, whereas streaming inference relies on generated motion for both, resulting in a training-inference mismatch. To reduce this history discrepancy, we introduce rehearsal training, which combines full-sequence teacher-forced supervision with supervised prediction along short autoregressive self-rollouts. To keep spatial conditioning consistent with generated motion, we propose spatial feedback, which recomputes body-relative source locations from generated root poses during both rehearsal training and inference. On SAM, Sonic-AMG achieves state-of-the-art FID and R-precision. It further supports real-time online generation of coherent reactions across sound events and source locations. Project page: https://jingchengni.com/sonic-amg/.