Video diffusion achieves faster streaming with improved long-term accuracy
Self-Aligned Forcing: Streaming Video Diffusion with Differentiable Noisy History
Computer Vision and Pattern Recognition
Summary
Generating videos frame by frame is tricky because small errors add up over time, leading to less accurate and less smooth motion in longer videos. The authors found a way to keep track of past information with adjustable noise levels and let the system learn from its own mistakes during training. Their method, called Self-Aligned Forcing, speeds up training and produces better video quality with smoother motion in longer streams. This helps video generation run faster and more reliably on a single GPU or across multiple GPUs.
What this means in practice
- •For video streaming engineers: Improve streaming video generation systems by reducing long-term visual errors and increasing processing speed on GPUs.
- •For augmented reality developers: Create longer, higher-quality generated video sequences with smooth motion for real-time AR applications relying on video diffusion.
Authors
Weiqiang Wang, Zhuokun Chen, Yusheng Dai, Boying Li, Yi Zhang, Hossein Rahmani, Qiuhong Ke, Jianfei Cai
Abstract
Autoregressive video diffusion enables interactive streaming generation, but suffers from error accumulation over long rollouts. Self-rollout training reduces exposure bias, yet finite rollouts leave long-range drift unresolved. We observe that the noise level of the history key-value (K/V) representations trades visual quality against motion, and that restoring gradients through the history aligns causal training far more closely with bidirectional training. Motivated by these observations, we introduce Self-Aligned Forcing (SAF), a training scheme that aligns the history of each block with the noise level of the block being denoised. Specifically, the history is the K/V produced by preceding blocks at the same denoising stage, so all blocks at a stage can be denoised in a single forward pass under a causal mask. This keeps the noisy history differentiable, allowing future losses to optimize how it is encoded. SAF therefore avoids a separate no-gradient rollout and per-block timestep-zero recaching, training up to 1.8x faster than prior methods with lower memory. At inference, SAF achieves the highest single-GPU throughput among existing methods and keeps one history bank per stage for a multi-GPU pipeline, reaching 49.1 FPS on 4 GPUs. Experiments show superior long-horizon generation with a better balance between visual quality and motion. Project page: https://anonymous.4open.science/w/self-aligned-forcing/.