Video tokenizer captures 4D world for compact video representation
VideoTok4D: A 4D-Aware Video Tokenizer for Compact World Representation
Computer Vision and Pattern Recognition
Summary
Current methods that turn videos into simple codes usually look at videos as a series of flat images and miss how things really move and change in 3D space over time. The authors propose VideoTok4D, a new way to break down videos into tokens that understand both the still parts and the moving parts in 3D and time. This approach makes video data much smaller while still keeping important details about how the scene looks and moves. They also use a special method to generate new 4D scenes efficiently from these tokens.
What this means in practice
- •For video compression engineers: Create highly compact video representations capturing 3D motion for better storage and transmission efficiency.
- •For graphics software developers: Generate realistic dynamic 3D scenes efficiently by using latent tokens that represent video content holistically.
Authors
Xinyi Chen, Hanxin Zhu, Xijun Wang, Xingrui Wang, Sen Liang, Xin Li, Zhibo Chen
Abstract
Video tokenizers have emerged as a cornerstone of modern video modeling, underpinning progress in compression, reconstruction and generation by mapping high-dimensional visual signals into compact latent spaces. However, despite this progress, current tokenization paradigms largely remain within the 2D visual domain, treating videos as image sequences rather than observations of an underlying dynamic 3D world. Consequently, the learned tokens inherit this observation-centric bias, limiting their capacity to compactly represent real-world 4D scenes. To mitigate this issue, we propose VideoTok4D, a novel 4D-aware video tokenizer for compact world representation. Specifically, our approach comprises three key designs: 1) a spatiotemporal disentanglement strategy that factorizes videos into static and dynamic tokens for holistic world modeling; 2) a track-aware dynamic attention mechanism that aggregates trajectory-aligned cues to promote cross-view motion consistency; and 3) Co4DGen, a diffusion prior learned over the resulting VideoTok4D token space for efficient 4D scene generation. Extensive experiments have demonstrated that our proposed method achieves state-of-the-art performance while requiring up to 4 orders of magnitude less storage than dense 4D representations. Moreover, the compact token space substantially shortens diffusion sequences, enabling efficient generation.