Diffusion model speeds up real-time streaming video enhancement
ReCaVSR: One-Step Streaming Diffusion Video Super-Resolution with Recycled Latents and Learned Cache Routing
Computer Vision and Pattern Recognition
Summary
Making videos look clearer and sharper as they play live is hard because it usually takes a long time to process each frame. The authors developed a method called ReCaVSR that uses clever tricks to remember important video details from recent frames without needing to look back at everything. This approach makes the video clearer faster and keeps it looking natural and consistent over time. They tested it on real and fake video sets and found it works better and faster than previous methods.
What this means in practice
- •For video streaming engineers: Build live video enhancement tools that improve quality with lower latency and memory on GPUs using recycled latent states and learned caching.$Commercial implications: Enables faster and less memory-intensive live video super-resolution services for streaming platforms, improving user viewing experience.
- •For gpu hardware designers: Design GPUs optimized for transformer caches and efficient decoding supporting fast real-time video super-resolution workloads like ReCaVSR.
Authors
Xijun Wang, Xin Li, Suhang Yao, Zirui Lang, Bingchen Li, Zhibo Chen
Abstract
Real-time diffusion-based video super-resolution (VSR) is in high demand for online streaming, yet stringent latency requirements often compromise generative fidelity. We propose ReCaVSR, a Wan2.2-based, one-step framework for streaming VSR that builds on two observations: recycled SR latents retain local temporal context, reducing the need for full historical Key-Value (KV) caches; and individual transformer layers benefit from distinct temporal scopes. ReCaVSR combines three complementary designs: (i) layer-wise cache routing with recycled SR latents: each DiT layer learns its KV-cache temporal scope under a cache budget and exports a static inference schedule, while recycled SR latents propagate local context by conditioning each new block on the model's own preceding predictions. (ii) Multi-Scope Query (MSQ) Discriminator: a compositional discriminator combining global, spatial-window, and temporal-tube feedback for holistic realism, local texture generation, and temporal stability. (iii) LR-conditioned adaptation of FlashDecoder: a VAE decoder that incorporates LR observations for efficient latent decoding. ReCaVSR enables streaming VSR without iterative sampling or full historical KV-cache materialization. Experiments on synthetic and real-world VSR benchmarks show better perceptual quality, temporal consistency, and streaming efficiency than representative VSR baselines. At $1080{\times}1920$ output resolution on a single NVIDIA A100-80GB, ReCaVSR achieves 21.20 FPS with 15.16 GB peak allocated GPU memory, running 2.72$\times$ faster while using 38.0\% less peak allocated memory than FlashVSR Tiny. The code is available at https://github.com/kopperx/ReCaVSR.