Streaming video editing runs faster with a new adaptation method
Streaming Video Editing with Easy Adaptation
Computer Vision and Pattern Recognition
Summary
Editing videos as they play, without waiting to load the whole video, is tricky because each video frame depends on others. The authors looked at existing methods and found ways to separate features and treat frames independently to make streaming editing work better. They created a new approach that uses a special model to control video edits in real time, training it so the system can quickly adapt without losing quality. Their method, called SVEET, edits videos smoothly and fast, running at 15 frames per second on powerful hardware.
What this means in practice
- •For video production teams: Enable real-time editing of high-quality videos during streaming without pre-processing the entire footage.
- •For live broadcast engineers: Integrate controllable, low-latency video effects and modifications directly into live video streams using trained diffusion models.
Authors
Yujia Hu, Jiajun Li, Zihao He, Songhua Liu
Abstract
In this paper, we propose SVEET, a framework that requires merely training on a pretrained bidirectional video diffusion model but supports high-quality streaming video editing in an auto-regressive fashion. To tackle this problem, we first systematically revisit existing video-to-video diffusion approaches and identify two key principles for such streaming adaptation: backbone feature disentanglement and conditional frame independence. Building on these insights, we develop a novel paradigm for controllable video generation. At its core, an auxiliary model branch encodes source video inputs with temporally independent self-attention, and the intermediate features are injected into the corresponding backbone blocks for streaming-compatible control. Moreover, to bridge the discrepancy between the feature spaces of bidirectional and streaming models, we propose a decoupled training scheme that explicitly enforces the orthogonality between the optimization directions of video controllability and model causality. Such disentanglement ensures compatibility between the two objectives at inference and facilitates smooth zero-shot knowledge transfer across heterogeneous backbone architectures. Extensive experiments demonstrate that SVEET achieves superior editing quality while maintaining real-time performance, attaining 15 FPS on a single H100 GPU 17 without any auxiliary acceleration techniques. Codes are available at https://github.com/YujiaHu1109/SVEET.