Uni voe enables better video object removal and insertion with interaction effects
UniVOE: A Removal-Driven Framework for Interaction-Aware Universal Video Object Editing
Computer Vision and Pattern Recognition
Summary
Removing or adding objects in videos is tricky because objects often interact with their surroundings in complex ways. The authors developed UniVOE, which first learns to remove objects and their effects from videos, then uses this to teach a model how to add objects back realistically. This method creates training data automatically, improving video editing quality without needing hard-to-get real paired videos. They also use a feedback system that aligns the results with what people prefer to see. Tests show UniVOE works better than existing tools, especially when objects interact with their environment.
What this means in practice
- •For video content creators: Edit videos to remove or add objects realistically, preserving interactions with the scene without needing extensive paired training data.$Commercial implications: Enables advanced video editing tools that deliver natural results, suitable for professional video production software.
- •For augmented reality developers: Create AR experiences where virtual objects interact believably with real-world video by learning from removal-driven object insertion.
Authors
Shuzhou Yang, Lanqing Guo, Lingen Li, Weiqi Li, Zhao Zhong, Liefeng Bo, Jian Zhang
Abstract
Video object removal and injection are reciprocal editing operations between object-present and object-absent video states, yet they differ substantially in training difficulty. Removal can be effectively bootstrapped from video inpainting and further adapted to erase object-related interaction effects. Injection, however, requires large-scale paired supervision to model object appearance, dynamics, and scene-dependent effects, while perfectly aligned real-world pairs are difficult to collect and synthetic data lacks sufficient realism and diversity. To address this challenge, we present UniVOE, a removal-driven framework for training both video object removal and injection models. We first train an effect-aware remover by transferring video inpainting priors and finetuning on object-present/object-absent pairs. The trained remover converts real object-present videos into object-absent counterparts, creating large-scale paired supervision for injection. An injection model is trained on these pairs in the reverse direction to recover objects together with their dynamics and interaction effects. Moreover, since pixel-level reconstruction alone cannot adequately capture perceptual realism, we introduce a reward-based post-training method that aligns injection results with human preferences using a proprietary reward model. Experiments against strong open-source and commercial systems demonstrate that UniVOE achieves state-of-the-art performance on both object removal and injection, particularly in challenging interaction-effect cases.