Soundwich generates video with separate editable audio tracks
Soundwich: Video Generation with Layered and Controllable Audio
Computer Vision and Pattern RecognitionMultimediaSound
Summary
Creating videos with matching sounds is common, but usually the sounds come as one mixed track, making it hard to edit parts like speech or music separately. The authors made Soundwich, a tool that splits sounds into separate tracks for speech, music, effects, and more, all perfectly timed with the video. It keeps these sound tracks connected so they make sense together, yet lets you change, mute, or move any part on its own. This approach helps people work with video and sound like in real studios, letting creative editing while keeping everything synced.
What this means in practice
- •For video editors: Create videos with separate, synchronized audio tracks independently editable for speech, music, and effects.
- •For game sound designers: Generate multiple synchronized sound layers matching visuals that can be edited separately for dynamic game audio.
Authors
Zhuo Ning, AmirHossein Naghi Razlighi, Sagi Polaczek, Daniel Cohen-Or, Ali Mahdavi-Amiri
Abstract
Recent joint audio-video generative models can synthesize realistic videos with synchronized sound, but typically generate audio as a single mixed track. This limits source-level control and differs from practical audiovisual workflows, where speech, music, sound effects, and ambient sounds are represented as separate editable tracks. We introduce Soundwich, a training-free framework that transforms a frozen joint audio-video flow-matching model into a generator of multiple synchronized, independently editable audio stems coupled to a shared video. Soundwich generates separate audio stems with explicit control over their temporal activity. To keep separately generated sounds coherent, we introduce a shared scene representation that communicates global audiovisual context across stems while preserving their source-level separation. We further route cross-modal interactions between each audio stem and its corresponding visual source, improving audiovisual consistency. The resulting stems remain synchronized with the video and can be independently retimed, muted, replaced, or remixed. Experiments and human evaluations show improved temporal control, source separation, and naturalness, while enabling flexible source-level editing within coherent audiovisual generation. Code is available at https://github.com/CodyNing/Soundwich.