CrossEdit enables flexible editing across images audio and video
CrossEdit: Cross-Modal Training Enables Rich Audio-Visual Editing
Sound
Summary
Editing multimedia content like pictures, sound, and video often requires changing only certain parts based on detailed instructions. Existing tools usually handle simple changes in just one type of content and struggle when sound and visuals must be edited together. The authors created CrossEdit, a single model trained to follow complex instructions and make edits across images, audio, and video without needing direct examples in every case. They used clever techniques to generate training data and showed that the model can apply what it learned from one type of media to others, improving editing quality on movies and other benchmarks.
What this means in practice
- •For video editors: Perform complex edits on movie scenes that combine audio and visual changes without needing separate tools for each media type.
- •For podcast producers: Edit speech and background sounds seamlessly while maintaining lip-sync accuracy in related video content.
Authors
William Chen, Prem Seetharaman, Ke Chen, Oriol Nieto, Kevin Duarte, Siddharth Srinivasan Iyer, Mamshad Nayeem Rizve, Zhiwen Cao, Shinji Watanabe, Yuanjun Xiong, Jianming Zhang, Zeyu Jin, Justin Salamon
Abstract
Multimedia editing requires generative models to interpret complex, compositional instructions and modify only the desired elements across modalities, a capability largely beyond existing systems. Current approaches are typically limited to simple, single-attribute edits within one modality, since diverse, high-quality editing pairs are hard to obtain, especially for cross-modal tasks where aligned audiovisual (AV) data is scarce. We present CrossEdit, a unified omni-modal editing model for images, audio, and video that performs AV movie scene edits zero-shot through cross-modal transfer. Our key observation is that complex instruction-following, once learned in any modality, generalizes to others. We exploit the additive nature of acoustic signals to procedurally generate large-scale audio editing pairs with compositional instructions, and show that fine-tuning on this synthetic data and self-supervised AV masked reconstruction, alongside a targeted set of curated cross-modal tasks, induces instruction-following that transfers zero-shot to unseen modality and instruction combinations. To evaluate this capability, we release CrossEditBench, a human-annotated benchmark of AV edits on movie scenes, and propose AV-FES, a metric that jointly scores instruction following and consistency. We show that our proposed techniques improve performance on zero-shot AV movie scene editing, while maintaining or improving performance on lip-synced speech editing and standard image, video, and audio editing benchmarks. Demos are available at https://wanchichen.github.io/crossedit/.