Connected self forcing improves long video prediction quality and consistency
Connected Self Forcing: Beyond Local Learning in Video Autoregression
Computer Vision and Pattern Recognition
Summary
Predicting long videos frame by frame can be tricky because errors add up, making videos look worse and less smooth over time. The authors introduce Connected Self Forcing, a way to train video prediction models that lets later predicted frames influence how earlier frames are generated, improving overall video quality and smoothness. They do this by reconnecting the mathematical paths that let the model learn from its own long-term predictions without using excessive memory. This method leads to better-looking videos without changing how the model works when making new predictions.
What this means in practice
- •For video streaming engineers: Enhance long video streaming quality by training prediction models that keep videos visually consistent and smooth over extended playback times.
- •For game developers: Improve in-game video or animation prediction to maintain visual quality during extended sequences, using connected gradient training techniques.
Authors
Dongbin Zhang, Chaoda Zheng, Kangjie Chen, Xiangyu Li, Shijia Chen, Jinhao Deng, Yuqi Zhang, Guangfeng Jiang, Hongbin Lin, Choo Sin Wai, Minqi Wang, Puyi Wang, Jingye Zhang, Yu Zhang, Xianming Liu, Boyang Wang
Abstract
To stream long videos while maintaining visual quality and temporal consistency, Self Forcing mitigates exposure bias through self-rollout training on self-generated histories with key-value (KV) caching. To keep memory manageable, it detaches historical caches, preserving forward dependencies between chunks but severing the backward gradient paths. We introduce Connected Self Forcing, a training framework that reconnects gradient paths across autoregressive chunks, allowing feedback from later predictions to guide how earlier context is generated. These connections go beyond historical KV-writing: gradients pass through generated latents into the computations that produced them, linking the generation of earlier context to its use in later predictions. To make this connected training memory-efficient, we develop shortcut gradient replay, which recovers cross-chunk gradients without retaining the full rollout computation graph. Integrated with distribution matching distillation, Connected Self Forcing trains historical chunks according to both their direct supervision and their contribution to subsequent generation. Experiments on autoregressive video generation show improvements in long-horizon visual quality and temporal consistency, without changing the inference procedure.