Discrete tokens match continuous visuals in multimodal AI understanding
Can Discrete Tokens Serve as Semantically Lossless Representations of Visual Signals?
Computer Vision and Pattern Recognition
Summary
Visual information can be turned into pieces called discrete tokens, like words in a sentence, to help computers understand images together with text. It's usually thought that these tokens lose important meaning compared to continuous image data. The authors found that more bits used to represent these tokens improve understanding and introduced a new way to split and encode image features that beats old methods. Their approach performs as well as continuous representations on complex tasks, including reading fine text in images, making discrete tokens a good option for future AI that combines images and language.
What this means in practice
- •For multimodal ai developers: Create AI models that analyze images and text together using discrete tokens without losing detailed visual meaning.
- •For document digitization teams: Improve optical character recognition on complex documents by using discrete token-based image representations that rival continuous methods.
- •For digital content platforms: Build more efficient AI-powered image and text processing tools for multimedia content using discrete visual tokens for better scalability.$Commercial implications: Enables novel commercial AI services that combine image and text understanding more efficiently for content providers.
Authors
Jialv Zou, Xiaotong Li, Zhengrong Yue, Xu Huang, Yaxin Luo, Wenhao Yan, Haodong Li, Wenyu Liu, Xinggang Wang
Abstract
Discrete visual tokens represent vision in a discrete form analogous to language, offering a native interface for multimodal modeling under a shared next-token prediction objective. However, quantization error is widely assumed to cause unavoidable semantic loss, fundamentally limiting the semantic fidelity of discrete visual tokens and positioning continuous visual representations as an intrinsic upper bound. Using multimodal understanding as a proxy for semantic fidelity, we revisit this assumption and find quantization error unreliable, as discrete representations with substantially larger errors can achieve better understanding. Instead, the effective bit budget of discrete indices emerges as a key determinant of downstream performance. However, existing quantizers eventually saturate as additional codebooks contribute progressively less semantic information. As one approach to alleviating this saturation, we introduce Factorized Vector Quantization, which quantizes complementary, fixed-dimensional views of the visual feature with symmetric parallel codebooks. It continues to improve at larger bit budgets, ultimately matching continuous visual representations in multimodal understanding. Trained on industrial-scale multimodal data, our model achieves state-of-the-art performance among discrete-representation models, outperforming UniAR and Emu3-Chat by 22.8% and 59.2%, respectively, on a nine-metric average. To our knowledge, it is also the first discrete-representation model competitive with strong continuous-representation models such as Qwen3-VL, including on fine-grained OCR tasks. We further identify the key principles and a practical recipe enabling this competitiveness. These results and insights support the potential of discrete visual representations to underpin large-scale native multimodal foundation models. Code and models are available at https://github.com/hustvl/FVQ