SonoWorld4D creates immersive 4D audio visual scenes from single videos
SonoWorld4D: From One Video to a 4D Audio-Visual Scene via Diffusion Synchronization
Computer Vision and Pattern RecognitionSound
Summary
Generating dynamic scenes that include both visuals and realistic sound is difficult because sound depends on motion and position. The authors introduce a new task to turn a simple video into a 4D scene you can explore, including spatial audio that changes with your viewpoint. Their method recreates a full background, tracks moving sound sources, and synchronizes sounds physically with the scene’s geometry. This lets you see and hear events from new angles in real time.
What this means in practice
- •For virtual reality developers: Generate explorable dynamic 4D scenes with synchronized spatial sound from a single video input to enhance VR experiences.$Commercial implications: Enables VR applications and platforms to create immersive content with realistic spatial audio and visuals automatically from video data.
- •For game designers: Create dynamic audio-visual environments with physically grounded sound sources from real-world videos for interactive game settings.
Authors
Derong Jin, Xiyi Chen, Ming C. Lin, Ruohan Gao
Abstract
Recent progress in visual scene generation has extended beyond static 3D scenes toward the more challenging problem of generating dynamic 4D scenes from minimal input. However, sound, despite being intrinsically coupled with dynamic objects and events, remains largely unmodeled in prior work. We introduce Video-to-4DAVScene, a new task that turns a video into an explorable, time-varying 4D audio-visual scene, and present the SonoWorld4D framework for this task. Given a monocular video, our method generates a 360 static background, reconstructs explicit motion trajectories of dynamic, sound-emitting foreground objects, and synthesizes a physically-grounded sound field consistent across multiple sources through dual-space audio diffusion synchronization. The resulting 4D scene supports real-time novel-view visual and spatial audio rendering. Evaluation on real-world dynamic scenes and a user study demonstrates SonoWorld4D generates 4D audio-visual scenes with perceptually realistic, physically grounded spatial audio consistent with scene geometry and visual content.