Video model improves geometry consistency under camera motion
Learning to Correct Geometry in Generated Videos
Computer Vision and Pattern Recognition
Summary
Videos generated by AI often show strange distortions when the camera moves because the system doesn't learn how objects should look from different angles. The authors noticed this problem comes from lacking data about real camera movements. They created a method called Video Geometry Corrector that learns from fake but realistic video pairs where one is distorted and the other is correct. This model fixes geometric errors in AI-generated videos afterward, helping objects maintain realistic shapes as the camera moves.
What this means in practice
- •For video production teams: Improve the realism of AI-generated videos by automatically correcting geometric distortions caused by camera movement.
- •For augmented reality developers: Enhance virtual content integration by ensuring consistent geometry when simulating camera motion in generated video sequences.
Authors
Weijie Lyu, Tiancheng Shen, Jason Li, Yujing Wang, Xiaoyuan Wang, Haobo Yuan, Yizhou Zhao, Ming-Hsuan Yang
Abstract
Generated videos often lose geometric consistency under camera motion: structures warp under viewpoint change, parallax becomes inconsistent, and smooth camera movement may collapse into shot-like transitions. We argue that this failure is largely a data problem. In web-scale video corpora, most visible motion comes from dynamic objects captured by static or weakly moving cameras, leaving only sparse supervision for viewpoint-induced appearance changes. To address this, we propose Video Geometry Corrector (VGC), a post-hoc video-to-video model trained on synthetic paired supervision. Starting from real videos with diverse camera motion and pose annotations, we construct VG-Pairs with a synthetic paired-supervision data generation pipeline. Each video clip is treated as a geometry-consistent target and paired with a distorted counterpart synthesized by a pretrained image-to-video inpainting model using trajectory-adaptive keyframes. VGC learns to map the distorted input to the geometry-consistent target without modifying the source generator, and uses optical flow as an auxiliary cue to help disentangle camera motion from object motion. Experiments on pose-conditioned and prompt-driven benchmarks show that VGC improves geometric consistency across multiple source generators, and scaling to a larger backbone further extends the framework toward more general video refinement.