Video matting model improves smooth edges across long video clips
VideoMaMa++: Temporally Consistent Video Matting via Preserve-and-Refine Token
Computer Vision and Pattern Recognition
Summary
Video matting is a process that separates a foreground subject from the background in videos. Existing methods that use diffusion models perform well but only on short video segments, which causes flickering and mismatched edges when the video is longer. The authors propose a new approach that uses special tokens to carry information between video chunks, maintaining consistent and smooth edges across the whole video. Their method also combines detailed color textures with rough masks more flexibly, improving the quality and temporal consistency of the matting results.
What this means in practice
- •For video editors: Produce videos with clean and flicker-free foreground extraction over long scenes using improved matting techniques.$Commercial implications: Enables video editing software to offer higher-quality and more stable background removal for professional and consumer videos.
- •For augmented reality developers: Create smooth and temporally consistent object masks for AR effects over long video durations without flickering artifacts.
Authors
Sangbeom Lim, Seoung Wug Oh, Heeji Yoon, Seungryong Kim, Joon-Young Lee
Abstract
Diffusion-based video matting has recently emerged as a leading approach, leveraging video diffusion priors from internet-scale pretraining to achieve strong zero-shot generalization from limited synthetic data. However, the heavy compute of these diffusion backbones limits inference to a short chunk of just 6 to 24 frames, so even ordinary videos span multiple chunks. Because each chunk is matted independently from coarse mask guidance, existing methods fail to maintain consistency across chunk boundaries, producing flickering edges and discontinuous transparency. We address this with a mask-guided diffusion-matting framework built on two ideas. The RGB and mask conditioning are represented as separate token streams that interact through full spatio-temporal self-attention, letting the model adaptively combine fine RGB texture with coarse mask layout rather than committing to a fixed channel-wise fusion. On top of this, we propose a novel pair of learnable embeddings, the preserve token and the refine token, that act as a per-frame conditioning interface and enable a sliding-window inference scheme in which already-generated mattes are propagated across chunk boundaries for local temporal consistency, complemented by a globally shared reference anchor at every chunk to prevent error accumulation in long sequences without any additional training. Across standard video matting benchmarks and in-the-wild long footage, our framework outperforms prior diffusion-based methods in both fine-detail accuracy and temporal consistency, and uniquely maintains consistency across chunk boundaries on long videos.