V-RAE: Rethinking Video Latent Spaces for Generation
2026-08-13 • Computer Vision and Pattern Recognition
Computer Vision and Pattern Recognition
AI summaryⓘ
The authors developed V-RAE, a video autoencoder that uses fixed high-level vision features instead of focusing just on pixel accuracy. Their model compresses video data by removing redundant frames while keeping important meaning, which helps generate and predict videos better. They tested V-RAE with different encoders and showed it outperforms previous methods in video reconstruction and generation, while learning faster. They also introduced a new way to measure how smooth and coherent generated videos are over time, which better reflects quality than traditional metrics.
video autoencoderlatent spacegenerative modelingvision foundation modelstemporal poolingvideo reconstructionclass-conditional generationfuture video predictiontemporal coherenceevaluation metrics
Authors
Minghui Guo, Shengqiong Wu, Hao Fei
Abstract
Latent video generation relies on autoencoders to define a compact space in which generative models operate. Although video autoencoder architectures have evolved substantially, their latent spaces are still optimized primarily for pixel-level reconstruction and provide limited high-level semantic organization. A reconstruction-optimal latent space, however, need not be well suited to generative modeling. We propose V-RAE, a video representation autoencoder that builds compact generative latents on top of frozen vision foundation model representations. A lightweight temporal pooling module removes temporal redundancy while preserving semantic structure, and a video decoder reconstructs continuous motion from the compressed features. We evaluate V-RAE with four representative frozen encoders on video reconstruction, semantic probing, and class-conditional generation. V-RAE achieves 2.13 rFVD on K600, outperforming all evaluated large-scale pretrained video VAEs. Its latents retain substantially more semantic information than conventional video tokenizer latents. Under matched generation settings, our best variant achieves gFVD scores of 117.86 and 19.16 on UCF101 and K600, respectively, while converging up to 6x faster}. We further show that reconstruction quality alone is insufficient to characterize generative utility and introduce tFVD, a temporal-coherence diagnostic that correlates more reliably with downstream generation quality. Beyond video generation, V-RAE also improves future video prediction on Cityscapes over the Wan 2.2 VAE latent space under matched prediction settings. Taken together, the experiments show that frozen semantic representations can support video reconstruction, generation, and predictive modeling. The project page: https://v-rae.github.io/.