Immersive audiovisual experience created from silent narrow videos
Enabling Immersive Audio-Visual Experience from Any Video
SoundMultimedia
Summary
Most videos show only a small part of a scene and have no sound that matches where things are happening, which makes them less real. The authors developed OmniDream, a system that can take a silent video and create a full surround visual and sound experience where you can look around and hear sounds coming from the right places. It separates each sound source's unique audio from how the environment changes it, so it can play sounds realistically as if they were part of the expanded scene. Their tests showed this method makes the sound and visuals match better and feel more immersive compared to other methods.
What this means in practice
- •For virtual reality developers: Add spatially accurate sound to monocular videos converted into panoramic views to enhance immersion in virtual reality experiences.$Commercial implications: This enables VR content developers to create immersive audiovisual environments from existing silent videos, offering new products for VR platforms.
- •For video content creators: Enhance narrow field videos with spatial soundscapes matching expanded views to improve viewer engagement and realism.
Authors
Zitong Lan, Mutian Tong, Jiatao Gu, Mingmin Zhao
Abstract
Most videos capture only a narrow field of view and provide no spatial audio, limiting the sense of immersion they can provide. Recent video generation models can expand perspective videos into panoramic ones, but do not provide the corresponding spatial soundscape. Without spatially consistent audio, these expanded visual worlds remain incomplete. This paper presents OmniDream, a training-free framework that transforms a silent monocular video into an immersive audiovisual experience, where viewers can freely look around while sounds remain spatially aligned with the visual scene. At the core of OmniDream is an object-centric audio representation that disentangles each sound source's intrinsic audio content from its scene-dependent acoustic effects, enabling independent audio generation, physics-based simulation of propagation effects, and flexible spatial audio rendering. Experiments show improved audio-visual alignment, spatial correctness, and perceptual immersiveness over baselines. Examples are available on https://huggingface.co/spaces/CuriousAlien000/spatial-audio-360-demo