Unified model improves audio and video creation and editing together
UniAVC: Unified Audio-Video Creation via Progressive Training and Asymmetric Scheduling
Computer Vision and Pattern RecognitionMultimediaSound
Summary
Creating videos with matching sound and editing them together is hard because data that pairs audio and video is scarce. The authors introduce a way to train a single model that first learns audio and video separately from large datasets, then learns how they relate using paired data. They also design separate processes that treat audio and video differently during generation, so each can be made or edited while keeping the other part intact. This approach lets the model handle various tasks like making both audio and video from text, converting one to the other, and editing one using the other’s cues, all with the same program.
What this means in practice
- •For multimedia developers: Build apps that create synchronized audio and video together from text descriptions and allow editing one part by referencing the other.
- •For audio engineers: Automatically generate or refine audio tracks conditioned on existing video content to improve editing efficiency without rerunning the whole process.
Authors
Kien T. Pham, Yuhao Liu, Chong Hou Choi, Zhenqi He, Yanghao Wang, Chenyang Lei, Rui Liu, Qifeng Chen, Long Chen
Abstract
Unified audio-video creation requires a single model capable of text-conditioned joint synthesis, bidirectional cross-modal generation, and instruction-guided editing across both modalities. Existing methods rely heavily on paired corpora, which are scarce for joint generation and even rarer for editing. Consequently, they are spent learning single-branch generative and editing capabilities that abundant unimodal data could readily teach, rather than on the cross-modal coupling. Furthermore, prevailing systems apply a single shared noise schedule to both modalities, neglecting two critical limitations. First, due to the significant discrepancy in information density, video requires prolonged high noise to organize spatial and motion structure, whereas the sparser audio signal can settle earlier during the synthesis process, the dynamics a uniform schedule might compromise. Second, lockstep denoising leaves asymmetric workflows, such as conditioning on one modality to synthesize or edit another, untrained. We present UniAVC, which addresses both obstacles. It first trains each branch on unimodal dataset to full generative and editing competence, and subsequently dedicates paired data to exploit cross-modal correspondence. Modality-decoupled schedulers meanwhile warp the noise schedule independently from a shared timestep, while subtasks with one clean modality directly supervise asymmetric configurations. These settings respectively enable intent-aware editing, which preserves one modality intact while enlisting it to constrain the other, and post-inference self-refinement, which regenerates only an unsatisfactory branch against an accepted one. UniAVC thus spans multiple creation tasks from a single set of weights without task-specific adapters.