Image tokenizers influence text and image tasks in multimodal AI models
Studying Image Tokenizers as Visual Languages in Unified Multimodal Models
Computer Vision and Pattern RecognitionComputation and Language
Summary
Image tokenizers turn pictures into a kind of language that computers can understand alongside text. This paper studies how well different tokenizers work when images and words are learned together in an AI model. The authors found that how these tokenizers perform depends on the specific task and that better image reconstruction doesn't always mean better overall results. They also show that choosing the right image tokenizer can affect how well the model understands text when both are trained together. This helps improve how future AI handles pictures and words jointly.
What this means in practice
- •For multimodal ai developers: Choose or design image tokenizers to improve joint modeling of text and images in AI systems for better generation and understanding.
- •For ai model trainers: Use image-to-text prediction loss as a reliable metric to guide multimodal training progress and downstream task performance.
Authors
Siting Li, Zhengyang Wang, Simon Shaolei Du, Xi Chen, Yang Liu
Abstract
Image tokenizers define the ``visual language'' of unified multimodal models, yet are commonly studied through isolated metrics or generation-/understanding-only evaluations. These evaluations do not fully capture how visual tokens behave when modeled jointly with text. We build a controlled pure-autoregressive testbed and track task-specific validation losses during multimodal continual pretraining across text, image, text-to-image (T2I), and image-to-text (I2T) prediction. We examine how these losses scale and relate to downstream performance, then use them to study multimodal learnability---how well image and text tokens are jointly modeled---and tokenizer design. We find that (1) losses should be analyzed by task, since they exhibit distinct scaling behavior and rank tokenizers differently. (2) The loss--performance relationship depends on the predicted token space: for a fixed tokenizer, T2I and I2T losses correlate with generation quality, but across tokenizers, the T2I loss--performance relationship shifts with the image-token space, whereas I2T loss, computed over a shared text vocabulary, provides a more consistent signal. I2T loss also correlates with both generation and visual understanding performance after supervised finetuning. Using losses as a lens, we show that (3) better reconstruction does not necessarily yield lower task-specific losses or stronger downstream performance, and that (4) image tokenizer choice can affect text modeling under joint optimization. As case studies, we revisit three tokenizer design axes---the discriminator, semantic supervision, and vocabulary size---to examine their effects on joint modeling and downstream performance. Together, our testbed offers a complementary perspective on image tokenizers as visual languages, highlighting their interplay with text in joint multimodal training.