Summary
Face swapping in videos means replacing a person's face with another while keeping all other details like movements and expressions intact. The authors created a new method called StreamingSwap that improves how reliably the new face looks like the target identity and how smoothly it works over long videos. They introduced better ways to teach the system to focus on identity changes and to generate video frames efficiently for real-time use. They also built a large video dataset to train and test their method. This makes face swapping possible at 20 frames per second on high-end hardware, even for videos up to 10 minutes long.
What this means in practice
- •For visual effects artists: Create realistic face-swapped videos in real time for film and TV productions using long video streams with minimal identity loss.
- •For live streaming developers: Integrate efficient face swapping into live streams, maintaining identity accuracy and video quality at interactive frame rates.$Commercial implications: Enables commercial live-streaming apps to offer real-time face swapping as a novel interactive feature.
Authors
Zekai Luo, Yuwei Liang, Ruiqi Ma, Haoyu Huang, Xuehao Wang, Huanyi Zheng, Mingyu Liu, Linhao Zhong, Chenchen Jing, Yuling Xi, Hao Chen
Abstract
Video face swapping transfers the reference identity to a target video while preserving its other attributes and motion. Despite recent progress in reference-guided flow-based methods, jointly achieving faithful identity transfer, high fidelity, and real-time generation over long video streams remains challenging. To address this challenge, we rethink the task-specific structure of reference-guided video face swapping. First, face swapping changes only identity while preserving most visual content, yet flow-matching supervision is weakly sensitive to these changes. We therefore develop Joint Rec-ID Loss, which balances explicit identity supervision and latent reconstruction according to the noise level. Second, the target video already provides non-edited attributes and temporal dynamics, enabling generation with fewer spatial tokens and shorter temporal contexts through Face-RoI Generation and Short-Context Rollout Training. By integrating these designs, we propose StreamingSwap, a reference-guided flow-matching framework for real-time streaming video face swapping. We also introduce HumanFaceVid, comprising 1,000 hours of cinematic training data and evaluation sets of 200 short and 200 long videos. Experiments demonstrate that our method achieves competitive identity transfer and visual quality at 20 FPS on a single NVIDIA RTX 4090 GPU, with stable generation over videos up to 10 minutes long. Project webpage: https://aim-uofa.github.io/StreamingSwap/