Audio visual editing method improves lip sync in human videos

Edit in Sync: Unified End-to-End Audio-Visual Editing for Human-Centric Videos

Computer Vision and Pattern Recognition

Summary

Videos of people speaking need their lip movements to match the sounds exactly. Past methods treated video and audio separately, leading to mistakes like lips not syncing well with speech. The authors created a new all-in-one system that edits both sound and image together without cutting out parts of the image. They also made a large collection of paired videos and sound samples to train the system. This approach improves how natural and synchronized the edited videos look and sound.

What this means in practice

  • For film editors: Edit speaking scenes by seamlessly swapping or inserting characters and their voices with precise lip-sync and natural motion in one process.$Commercial implications: Video editing tools can sell advanced features for automated, high-quality audio-visual character manipulation in human-centric videos.
  • For virtual reality developers: Create realistic avatars with synchronized speech and facial motions that can be directly edited from reference video and audio in VR environments.

Authors

Ziye Yuan, Minghong Cai, Xu He, Ruchang Yao, Xiao Fu, Kaixin Ding, Ming Zhang, Xiangyu Du, Tianpeng Lan, Da Xie, Xiangyu Yue

Abstract

Human-centric audio-visual editing requires precise alignment of lip movements, gestures, and expressions with speech. Existing methods process inputs through independent video and audio pipelines, preventing cross-modal interaction between visual editing and audio dubbing. Many of these pipelines further rely on mask-based inpainting. As a result, they suffer from poor lip-audio synchronization, cumulative pipeline errors, and an inherent mask-size dilemma. We present AVE, the first unified end-to-end framework that jointly performs reference-guided person swapping, insertion, and deletion across both audio and video modalities without requiring spatial masks. To overcome data scarcity, we design a synthesis pipeline and introduce AVE-Dataset, the first audio-visual editing dataset to provide 100K pairs with appearance and voice references. By combining Synthetic-to-Real reconstruction with preliminary audio warm-up, our framework transcends synthetic pipeline limits and mitigates audio-video token imbalance. AVE outperforms modality-separate audio-visual editing pipelines in editing realism, temporal consistency, and audio-visual alignment.