Streamerase enables real-time video object removal and erasing
StreamErase: Towards Ultra-Fast Streaming Video Erasing
Computer Vision and Pattern Recognition
Summary
Removing unwanted objects from videos usually takes a long time and can't be done while the video plays. The authors created StreamErase, a method that can remove objects from videos almost instantly, working at up to 26 frames per second on one GPU. They do this by first training a strong model with full video context, then transferring that ability to a fast, one-directional streaming model that can handle very long or even infinite videos without slowing down. This technology lets users see the erasing happen live as they watch the video, making the process interactive and practical for many uses.
What this means in practice
- •For video editing software teams: Integrate real-time object removal features that let users interactively erase objects from streaming videos without delay.$Commercial implications: Enables new video editing tools offering instant visual effects removal at consumer or professional level.
- •For live broadcast production teams: Remove unwanted objects or distractions from ongoing live video feeds to improve visual quality in real time.
Authors
Zhuobai Dong, Xiaobing Tu, Xiantao Zhang, Jinkui Ren, Yinggui Wang, Yue Ma, Linfeng Zhang
Abstract
Current video object removal methods suffer from prohibitive latency (minutes per video) and non-causal processing that prevents real-time interaction. We present StreamErase, enabling sub-second latency with up to 26 FPS streaming generation on a single GPU. Our approach first augments a text-to-video model with mask input to generate high-quality videos that erase target objects and synthesize coherent backgrounds, but does not perform inference on the fly. As such, we progressively transfer the erasing capability from a powerful bidirectional foundation model to an efficient unidirectional streaming eraser, enabling long-horizon erasing without sacrificing visual fidelity. Several key challenges arise when generating videos over long, potentially infinite horizons: (1) bridging the domain gap from training on finite length and extrapolating to long horizons, (2) sustaining high quality by preventing error accumulation, and (3) maintaining fast inference, without increasing computational cost as the context grows. To address these challenges, we carefully designed sliding-window causal attention with attention sinks. By incorporating self-rollout with attention sinks and KV cache rolling during training, we simulate inference-time extrapolations with a fixed context window, enabling constant-speed generation of arbitrarily long videos. Finally, we establish a dedicated benchmark, LongErase, for streaming long-video object removal. Our model achieves strong performance compared to existing video object erasing models and generalizes well to real-world video scenarios, uniquely enabling infinite-length streaming. With StreamErase, users can input a mask and see the erasing results unfold in real-time, delivering a truly interactive experience.