Video tokenizer improves fast semantic and detailed video generation
SemanTok: Predictable Semantic Tokens for Efficient Autoregressive Video Generation
Computer Vision and Pattern RecognitionMachine Learning
Summary
Generating videos with computers often requires breaking videos into smaller parts called tokens, which represent either the overall scene or fine details. The authors designed SemanTok, a new method that creates tokens better aligned with the video's meaning at every stage. This makes the computer's video predictions both faster and clearer, even when the video shows unfamiliar scenes. SemanTok uses features from a fixed vision model to guide the token creation, improving generation quality without needing very large models.
What this means in practice
- •For video generation engineers: Create video generation systems that produce higher-quality videos with more accurate semantics using shorter token sequences for efficiency.
- •For game developers: Generate realistic and semantically meaningful video content for dynamic in-game scenes with reduced compute cost.
Authors
Mikhail Dereviannykh, Vikram Voleti, Simon Donne, Mallikarjun Byrasandra Ramalinga Reddy, Shimon Vainer, Mark Boss
Abstract
Recent video-based world models pair the scalability of autoregressive (AR) prediction with the visual quality of diffusion models. The choice of scene tokenizer is paramount for the optimal performance of each of these, both in terms of fidelity and semantics. Flexible-length, coarse-to-fine tokenizers yield exactly that: the first coarse tokens carry the clip's global semantics while later tokens further specify details. Existing flexible tokenizers only apply a representation-alignment (REPA) loss on early decoder hidden states, a target the decoder can partly meet from its noised input instead. We introduce SemanTok, a flexible video tokenizer that feeds frozen DINO features into its encoder and adds lightweight heads that reconstruct them from each retained token prefix alone. SemanTok achieves high semantic alignment and video fidelity at every AR model size: a 201M SemanTok AR model matches or beats a VideoFlexTok AR model $3.4\times$ its size, and larger SemanTok AR models further improve fidelity. It keeps semantic alignment on out-of-distribution classes and gives the decoder higher semantic alignment at every noise level, including pure noise. It performs well in both reconstruction and generation, and its short token prefixes are cheaper to predict and give better generation fidelity, with pixel detail deferred to later tokens.