MatAnyone-E speeds up high-resolution video matting by focusing memory use
MatAnyone-E: Sparse Memory Propagation for Efficient High-Resolution Video Matting
Computer Vision and Pattern Recognition
Summary
High-resolution video matting tries to separate people or objects from backgrounds while keeping tiny details like hair sharp and smooth over time. The authors improved an existing method by first working on low-resolution versions to find tricky areas that need more attention. Then, only those areas get the full high-resolution treatment, saving a lot of time and memory without losing quality. Their method can run much faster and use less GPU power, even for very detailed 4K videos.
What this means in practice
- •For video editing software developers: Enable faster, high-quality video matting for 4K content by reducing memory use and computation on critical frame regions only.$Commercial implications: Makes real-time high-resolution video background removal products more viable for consumer and professional editing software.
- •For streaming platform engineers: Improve real-time matting performance on live streams without requiring excessive GPU resources, enabling smoother user experiences.
Authors
Shuyun Wang, Shangchen Zhou, Peiqing Yang, Yuekun Dai, Xin Yu, Chen Change Loy
Abstract
High-resolution video matting requires preserving fine details, such as hair strands and soft transparency, while maintaining consistency across frames. Memory-based methods achieve strong matting quality by matching current-frame features with information stored from earlier frames, but this process becomes increasingly costly as resolution grows. We present MatAnyone-E, an efficient extension of MatAnyone 2 that focuses computation where it is most needed. Our approach first performs low-resolution propagation to identify uncertain regions that require closer inspection and the memory candidates most relevant to them. It then performs high-resolution matching only for these regions using the selected memory, while reusing low-resolution matching results elsewhere. This strategy reduces both the number of locations requiring detailed matching and the amount of memory searched, while preserving the information needed to produce a complete matte. We further reduce the cost of generating high-resolution mattes with a lightweight decoder designed around measured runtime bottlenecks and trained to reproduce the original decoder's representations. Experiments show that MatAnyone-E maintains matting quality comparable to MatAnyone 2 while achieving an estimated 7.4x inference speedup and reducing peak GPU memory usage to 7.6% of the usage of MatAnyone 2 at 4K. With compilation-enabled inference on a single NVIDIA H200 GPU, it reaches 73.07 FPS at 1080p and 28.62 FPS at 4K.