MixTok combines global and local tokens to improve image encoding
MixTok: Adaptive Visual Tokenization with Global 1D Tokens and Sparse 2D Refinement
Computer Vision and Pattern Recognition
Summary
Encoding images efficiently means representing important details without using too much space. The authors introduce MixTok, a new method that uses both a global summary of the whole image and extra detailed tokens in selected areas with complex content. This approach adapts to each image, adding detail tokens only where needed while keeping the total number of tokens low. Tests show that MixTok can reconstruct and generate images with quality comparable to other methods, but using fewer tokens.
What this means in practice
- •For computer vision engineers: Create efficient image encoding systems that balance detailed local features and global information with fewer tokens.
- •For image generation developers: Build image generation models that maintain high quality using mixed global and local token representations.
Authors
Chenge Li, Yefei He, Xinyu Liu, Feng Chen, Hong Zhou, Bohan Zhuang
Abstract
Efficient visual tokenization requires balancing representation compactness with the preservation of fine-grained image content. Conventional grid-based tokenizers allocate tokens uniformly across spatial locations despite non-uniform visual complexity, whereas compact 1D tokenizers aggregate global information but can struggle to preserve local details. We introduce MixTok, a hybrid visual tokenizer that combines a compact 1D global representation with sparse 2D tokens for local refinement. A content-adaptive router ranks candidate locations by refinement priority and determines the number of 2D tokens for each image under a target average token budget. At the highest-scoring locations, 2D token features replace the spatial features recovered from the 1D representation, and a shared decoder reconstructs the image from the resulting mixed features. On ImageNet 256 x 256, MixTok achieves highly competitive reconstruction and generation performance, with an rFID of 1.195 and a gFID of 1.95 at respective average budgets of approximately 128 and 131 content tokens per image.