Video object removal evaluation benchmark aligned with human perception
VOR-Bench: A Human Perception-Driven Benchmark for Video Object Removal
Computer Vision and Pattern RecognitionArtificial Intelligence
Summary
Removing objects from videos is tricky to evaluate because existing methods often don’t match what people think looks good. The authors introduce VOR-Bench, a new way to test video object removal methods that better reflects how humans perceive quality. They created a diverse dataset including various types of videos and a system to produce realistic video pairs for testing. They also developed a new scoring model that aligns closely with human judgment on how well objects are removed. Their work helps make video editing tools that rely on removing objects more reliable and naturally evaluated.
What this means in practice
- •For video editing software developers: Evaluate and improve object removal features using a benchmark that matches human perception more closely.
- •For content moderation teams: Use improved evaluation tools to assess automated object removal in videos for compliance or privacy enforcement.
Authors
Haonan Huang, Tianrui Qiu, Xianghao Zang, Yinan Du, Zhixiang He, Chi Zhang, Hao Sun, Zhongjiang He, Tianwei Cao, Xuchong Zhang, Hongbin Sun, Kongming Liang, Zhanyu Ma
Abstract
Despite its crucial role in video object removal (VOR), existing evaluation paradigms face two critical limitations: questionable references and a misalignment between tradi- tional metrics and human preference. To address these challenges, we introduce VOR- Bench, which advances VOR evaluation through three integrated components. First, we present the VOR Dataset (VORD), the first benchmark dataset providing both paired edited videos and graffiti masks. Its unique strength lies in a diverse data spectrum, which encompasses model-generated, tool-rendered, and camera-captured data, ensuring robust assessment across real-world scenarios. Second, we develop rMPAF, a realistic Motion- capable Paired-video Acquisition Framework. By combining the strengths of image- based object removal and fine-tuned video generation models, rMPAF automatically generates realistic, motion-coherent paired videos. Finally, we propose three evaluation dimensions and introduce VOR-MDSM, the first perception-driven VLM-based scoring model specifically designed for mask-guided VOR. It bridges the gap between arithmetic metrics and human perception by covering the essential visual attributes and matching nuanced human judgment. Extensive experiments demonstrate that VOR-Bench yields evaluation results that align closely with human perception, achieving a remarkable cor- relation (\r{ho} > 0.9) with subjective assessments. We will release VOR-Bench along with its documentation to ensure full reproducibility.