Video frame prediction improved by aligning motion and semantics
Destination-Conditioned Adaptation with Semantic-Geometric Alignment for Generative Inbetweening
Computer Vision and Pattern Recognition
Summary
Making smooth videos by filling in the frames between a start and end image is hard, especially when objects move a lot or change shape. The authors propose a method that helps a video generation AI better follow its target ending frame, while keeping the motion realistic and consistent. They do this by teaching the AI to understand and align both the meanings and movements in the frames as it creates the in-between frames. Their approach reduces errors where the video drifts away from the target or moves oddly, improving video smoothness and quality.
What this means in practice
- •For animation studios: Generate smooth intermediate animation frames that accurately match the intended keyframes even with complex motion.$Commercial implications: It enables production-quality frame interpolation that can be sold as a software feature to animation companies.
- •For video game developers: Create realistic transitional frames for character movements between key poses to enhance visual fluidity in games.
Authors
Zhiyu Xu, Wei Shang, Yushen Xu, Wenwen Li, Zhiyan Xu, Pengfei Zhu, Dongwei Ren
Abstract
Generative inbetweening aims to synthesize temporally coherent intermediate frames between a start frame and a target frame. Although pretrained image-to-video diffusion models provide strong priors for large-motion inbetweening, they are not explicitly designed to reach a specified target frame. They may therefore generate realistic individual frames while drifting from the target or producing unstable motion under large displacements and non-rigid dynamics. We propose DSG-GI, a framework that adapts pretrained latent video diffusion models through \textbf{D}estination-Conditioned Adaptation and \textbf{S}emantic-\textbf{G}eometric Alignment. Destination-Conditioned Adaptation treats the target frame as the destination of the sequence and provides temporally progressive guidance along the generated trajectory. Semantic-Geometric Alignment aligns the token-wise spatio-temporal relational geometry of backbone features with that of a frozen video foundation encoder and applies a flow-matching-aware latent flow constraint to samples near the data endpoint of the probability path. Together, these components strengthen destination guidance and preserve coherent semantic-geometric motion structure. Extensive experiments on standard and large-motion inbetweening benchmarks demonstrate improvements in temporal stability, target-frame faithfulness, and perceptual quality. Our code will be made publicly available.