Video object removal methods leave visible content unused in videos
Recoverability-Aware Evaluation of Diffusion Video Inpainting and Object Removal
Computer Vision and Pattern Recognition
Summary
When removing objects from videos, some methods try to recreate the background behind the object by either copying visible content from other frames or making a new guess. The authors show that existing evaluations mix these two approaches, making it unclear which method works better. They propose a way to evaluate performance separately on parts of the video where background is visible and where it is not. They also introduce a method that blends copied colors from visible frames with diffusion-based video editing to reuse observed content, but found that current methods often leave such observed content unused.
What this means in practice
- •For video editing software developers: Improve object removal tools by separately evaluating and reusing visible background content to enhance video quality and temporal consistency.$Commercial implications: Enables the development of more accurate and reliable object removal features in video editing products by improving evaluation and reuse of visible content.
- •For visual effects artists: Use recoverability-aware evaluation to better understand when video object removal relies on copying versus generating content, guiding manual corrections in post-production.
Authors
Yanliang Qi, Jiacheng Ding, Cong Guo, Peihua Mai
Abstract
Video inpainting for object removal is scored over the whole hole, although the background behind the object is visible in other frames of the shot for some hole pixels and in no frame for others. A whole-hole score therefore mixes copying with generation and cannot show which of the two a remover does well. To separate the two, we propose recoverability-stratified evaluation, in which a point tracker run on the clean video labels each hole pixel by whether and how far away it is observed, so that fidelity and temporal stability can be reported per stratum against a copy reference. To reuse the observed content, we also propose evidence-guided sampling (EGS), which blends tracked colors into the clean-latent prediction of a video diffusion remover without training. On DAVIS, the perceptual advantage of three diffusion removers over flow propagation shrinks where the content is visible a few frames away (LPIPS difference-in-differences +0.046 to +0.056 against E2FGVI-HQ), and real paired captures show the same pattern for two of them. Their DAVIS hole warping error is two to four times that of ProPainter, and copying along clean-video tracks lowers their recoverable LPIPS by a quarter to two fifths. EGS closes 43% and 56% of the gap to this copy reference on held-out DAVIS when its colors come from clean-video tracks. With colors tracked on the masked input, a pre-registered confirmation finds no gain on MiniMax-Remover and a loss on ROSE. Diffusion removers thus leave observed content unused on DAVIS, and training-free reuse does not yet hold with tracked evidence.