Video generation method improves long-term smoothness and runs in real time
Consistent Video-to-Video Translation via Explicit Correspondences
Computer Vision and Pattern RecognitionArtificial IntelligenceGraphicsMachine Learning
Summary
Creating videos that change style or content frame by frame can be tricky if the changes don't look consistent over time. Existing methods focus only on short sections of the video, which can cause jumps or glitches when keeping track of what happened earlier. The authors show a way to look directly at which parts of earlier frames match up with the current frame and use this to keep the video changes smooth and consistent over the whole clip. This method still works quickly enough for interactive uses. Their experiments show it improves how smooth and consistent videos look without slowing things down.
What this means in practice
- •For video game developers: Maintain smooth and consistent in-game video effects that adapt instantly to player actions over extended scenes.
- •For video editing software teams: Provide users with real-time video style transfer tools that keep consistent details throughout long clips.$Commercial implications: Enables commercially viable interactive video editing features with improved quality for professional and consumer software.
Authors
Gaurav Parmar, Zhengqi Li, Richard Zhang, Jun-Yan Zhu, Srinivasa Narasimhan, Eli Shechtman, Yotam Nitzan
Abstract
Interactive video-to-video applications require real-time generation while maintaining long-range temporal consistency. However, recent methods achieve speed by restricting attention to a fixed-length causal window. This limits long-range consistency whenever relevant history falls outside the context window. To address this limitation, we introduce vid2vid-long, a correspondence-based approach grounded in a simple observation. Video-to-video translation provides an explicit cross-frame reference: the regions that match in the input video should match in the output video too. Concretely, we compute patch-level correspondences in the input, guiding retrieval of previously generated outputs in the corresponding regions, and condition the model on these patches through correspondence cross-attention. Our experiments demonstrate that vid2vid-long consistently improves long-range temporal consistency across recent video models while preserving generation quality and real-time performance.