StereoBind improves sound location with video source motion tracking
Here the World in Stereo: Learning Dynamic Spatial Correspondence for Immersive Joint Video-Audio Generation
Computer Vision and Pattern Recognition
Summary
Sounds in videos usually come from specific places and move along with objects seen on screen, like a car moving from left to right. Current video-audio tools can sync sounds and images but often miss the way sound location changes as the object moves. The authors of this paper created StereoBind, a system that tracks the movement of visual sources and matches stereo sound changes accordingly, making the experience feel more real and immersive. They also made a big dataset to help train and test their system.
What this means in practice
- •For augmented reality developers: Generate stereo sounds that accurately follow moving visual objects to enhance immersive AR experiences.$Commercial implications: Enables AR products with realistic spatial audio that matches user views, improving user immersion and experience.
- •For interactive game designers: Create stereo audio that dynamically changes with character or object motions for more convincing game soundscapes.
Authors
Hanmo Chen, Chengcheng Liu, Tianxiao Chen, Zheyu Zhang, Siming Zheng, Jinwei Chen, Xu Yang, Cheng Deng, Bo Li, Peng-tao Jiang
Abstract
Recent joint video-audio generation models have achieved strong semantic correspondence and temporal synchronization. However, applications such as AR/VR and interactive gaming further require stereo audio to provide an immersive sense, which remains largely overlooked. Effective stereo audio requires the perceived sound location to evolve consistently with the motion of its corresponding visual source. We refer to this property as Dynamic Spatial Correspondence and propose StereoBind, a framework that binds visual source motion to stereo sound generation. StereoBind uses motion tracks to coordinate visual motion and stereo audio through three complementary mechanisms. Visual Motion Binding establishes source-aware audiovisual correspondence, the Spatial Track Encoder captures absolute source positions, and Residual Track RoPE models relative motion. For supervision and evaluation, we construct StereoWorld-29K, a large-scale stereo audio-video dataset with paired motion tracks, and StereoWorldBench for measuring audiovisual spatial consistency. Experiments show that StereoBind substantially improves spatial alignment in stereo audio generation over existing models while preserving overall audiovisual quality.