Coarse videos converted to realistic ones without paired training data
Pair-Free Coarse-to-Real Video Generation via Real-to-Coarse Image Supervision
Computer Vision and Pattern Recognition
Summary
Making realistic videos from simple animations or game footage usually needs expensive matching pairs of rough and real videos for training. The authors found it’s easier to make pairs of real images and their rough versions, so they trained a model to turn real videos into fake rough videos. Then they used this to train another model that can convert real coarse videos into realistic ones without needing paired video data. This approach lets the training use any collection of real videos, not just synthetic paired ones.
What this means in practice
- •For visual effects artists: Generate photorealistic videos from coarse animations or game renders without expensive paired training data.
- •For game developers: Convert game-rendered videos into realistic footage automatically, improving visual quality without needing synthetic data pairs.
Authors
Gonzalo Gómez-Nogales, Daohan Lu, Peiye Zhuang, Tinghui Zhou, Yi Zhou
Abstract
Coarse-to-real (C2R) video generation converts coarse animation, simulation, or game-rendered videos into photorealistic videos while preserving scene layout, camera motion, and dynamics. Training such models requires aligned (coarse, realistic) video pairs, which are expensive to create and confine prior systems to narrow synthetic domains. We exploit a simple asymmetry: paired videos are hard to obtain, but paired real-to-coarse images are easy to construct with existing editing and 3D reconstruction tools. Using these image pairs, we train the DINO-R2C model to transfer realistic images to coarse images in DINO feature space, and apply it frame-wise to ordinary real videos to synthesize pseudo-coarse videos. The C2R# model then learns to generate each real video from its pseudo-coarse counterpart, free from needing (coarse, realistic) video pairs. At inference, the DINO-R2C model is discarded and actual coarse videos drive the C2R# model directly. Thus, our pair-free training scales with any real-video corpora rather than with paired synthetic videos. We will release the model and project webpage at https://gonzalognogales.github.io/coarse2real-sharp/.