Image tokenizers influence multimodal AI performance and training balance
Studying Image Tokenizers as Visual Languages in Unified Multimodal Models
Computer Vision and Pattern RecognitionComputation and Language
Summary
Understanding how images and text are combined in AI models depends a lot on how images are turned into 'visual words' called tokens. This paper studies how these image tokens work together with text tokens during training, revealing that different tasks and token designs affect how well multimodal models learn and perform. The researchers found that better image reconstruction doesn't always lead to better task results, and the choice of image tokenizer can even influence how text is understood by the model. Their work offers new ways to evaluate and design image tokenizers for AI systems that handle both pictures and text.
image tokenizermultimodal modelsautoregressive trainingvisual tokenstext-to-image generationimage-to-text predictionloss scalingvocabulary sizesemantic supervisionjoint optimization
Authors
Siting Li, Zhengyang Wang, Simon Shaolei Du, Xi Chen, Yang Liu
Abstract
Image tokenizers define the ``visual language'' of unified multimodal models, yet are commonly studied through isolated metrics or generation-/understanding-only evaluations. These evaluations do not fully capture how visual tokens behave when modeled jointly with text. We build a controlled pure-autoregressive testbed and track task-specific validation losses during multimodal continual pretraining across text, image, text-to-image (T2I), and image-to-text (I2T) prediction. We examine how these losses scale and relate to downstream performance, then use them to study multimodal learnability---how well image and text tokens are jointly modeled---and tokenizer design. We find that (1) losses should be analyzed by task, since they exhibit distinct scaling behavior and rank tokenizers differently. (2) The loss--performance relationship depends on the predicted token space: for a fixed tokenizer, T2I and I2T losses correlate with generation quality, but across tokenizers, the T2I loss--performance relationship shifts with the image-token space, whereas I2T loss, computed over a shared text vocabulary, provides a more consistent signal. I2T loss also correlates with both generation and visual understanding performance after supervised finetuning. Using losses as a lens, we show that (3) better reconstruction does not necessarily yield lower task-specific losses or stronger downstream performance, and that (4) image tokenizer choice can affect text modeling under joint optimization. As case studies, we revisit three tokenizer design axes---the discriminator, semantic supervision, and vocabulary size---to examine their effects on joint modeling and downstream performance. Together, our testbed offers a complementary perspective on image tokenizers as visual languages, highlighting their interplay with text in joint multimodal training.