Video object segmentation struggles with brief object appearances
When Predicting Nothing Beats SAM 3: Revisiting Evaluation in Video Object Segmentation
Computer Vision and Pattern Recognition
Summary
Video object segmentation tries to track objects throughout videos. The authors found that current tests mostly look at videos where objects are visible almost all the time and miss cases where objects appear briefly or intermittently. They created a new test called FaVOS to check how well segmentation methods perform when objects show up only some of the time. They found that existing ways to judge performance can be tricked by guessing no object is present, even when that’s not helpful. To fix this, they proposed a new scoring method that better measures the quality of tracking over time.
What this means in practice
- •For video analytics developers: Evaluate and improve video object tracking algorithms in scenarios where objects appear only briefly or sporadically in long videos.
- •For video surveillance teams: Better measure detection accuracy of objects that appear intermittently in security footage using improved evaluation metrics.
Authors
Jihwan Hong, Woohyeon Park, Jaeik Kim, Jaeyoung Do
Abstract
Video Object Segmentation (VOS) in complex and long videos is increasingly important for real-world applications, where target objects often appear only intermittently within long temporal horizons. However, existing benchmarks largely focus on temporally salient objects that remain visible for most of the video. To address this gap, we introduce FaVOS (A Benchmark for Video Object Segmentation with Fractional Temporal Visibility), a benchmark designed to evaluate VOS methods under low temporal visibility. We show that, in this regime, the standard J&F metric can collapse VOS evaluation into absence classification, because empty predictions receive high rewards on target-absent frames. Consequently, even a trivial empty-mask predictor can outperform strong models such as SAM 3, revealing a fundamental mismatch between current metrics and practical VOS performance. To mitigate this issue, we propose Volumetric J&F, which evaluates mask sequences as spatio-temporal volumes and reduces the dominance of target-absence rewards while preserving sensitivity to segmentation quality and temporal structure. Project page: https://aidaslab.github.io/FaVOS.