TrackTok creates stable object tokens for efficient video modeling
TrackTok: Object-Centric Video Tokenization with Semantically Persistent Tokens
Computer Vision and Pattern Recognition
Summary
Videos are made of many frames showing moving objects, and computers need to understand which parts belong to the same object over time. Existing methods break videos into fixed patches, which don’t track objects well as they move or change shape. The authors present TrackTok, a method that keeps tokens tied to the same objects through time, even if the objects move or get partially hidden. This reduces the number of tokens needed and helps make video generation models faster and more accurate.
What this means in practice
- •For video generation engineers: Build video generation models that use fewer tokens for faster synthesis without losing object details.
- •For computer vision developers: Create more efficient video frame prediction tools by tracking objects persistently across frames.
- •For animation software developers: Improve animation tools by enabling stable object tracking tokens for generating consistent object appearance over time.$Commercial implications: This enables selling advanced animation features that maintain object consistency in generated video sequences.
Authors
Nhi Pham, Christopher Wewer, Bernt Schiele, Jonas Fischer, Jan Eric Lenssen
Abstract
We introduce TrackTok, an efficient video tokenization framework that produces object-centric tokens, i.e., tokens that consistently encode the same semantic objects across time, even as it moves, deforms, or undergoes partial occlusion. Conventional patch-based or existing holistic tokenizers produce token identities often tied to fixed image locations. In contrast, our approach learns temporally persistent object-centric representations that bind visual features across frames into stable semantic units. This enables substantially more compact, continuous latent video representations, reducing the number of tokens required for downstream generative modeling with state-of-the-art flow models, while preserving the semantic structure needed for high-quality synthesis. Our tokenizer offers a favorable quality-efficiency trade-off. TrackTok achieves comparable class-conditional generative FVD on UCF-101 dataset at 8 times fewer latent tokens and better latency, and outperforms existing 1D tokenization baselines on frame prediction task on Kinetics-600 across low token budgets. These results suggest that object-aligned and temporally persistent tokenization is a promising direction for scalable video generation, where reducing latent token counts is critical for efficient modeling. Our code for this work is available at https://github.com/phamleyennhi/tracktok.