Mixture-of-Thought-Tokens: Unifying Perception and Reasoning for Free-form Multimodal Grounding
2026-07-27 • Computer Vision and Pattern Recognition
Computer Vision and Pattern Recognition
AI summaryⓘ
The authors address a challenge in AI models that try to understand images and language together, where current methods either struggle to precisely pinpoint objects or to think through complex questions about them. They propose a new method called Mixture-of-Thought-Tokens (Motto) that links special tokens directly to specific places in an image and allows the model to switch between identifying objects and reasoning about them. They also created a new benchmark called PR-Bench to test how well models handle both perception and reasoning. Their experiments show that Motto performs better than existing methods on various tasks where accurate localization and understanding are both needed.
Multimodal Large Language Modelsgroundingspatial localizationtokenizationchain-of-thoughtimage-language tasksreferring expression comprehensionvisual reasoningbenchmarkperception-reasoning gap
Authors
Tianyi Gao, Han Fang, Tianyi Ding, Hao Li, Xin Wei, Hongbo Sun, Xiaodong Dong, Ye Yuan, Jinglin Xu, Kongming Liang, Hao Sun, Jingmin Xin
Abstract
Multimodal Large Language Models have made great progress in grounding tasks, yet existing methods still struggle to unify precise localization and complex reasoning. For one thing, text-based methods rely on coordinates or index prediction, severely limiting the perceptual capabilities of the model for dense visual objects. Meanwhile, latent token-based methods employ special tokens without inherent spatial references and use a decoding mechanism that lacks thinking steps, weakening high-level reasoning capabilities. Consequently, developing a unified framework that excels in both perception and reasoning remains challenging. To address this, we propose Mixture-of-Thought-Tokens (Motto), a new free-form multimodal grounding method that bridges the perception-reasoning gap, enabling MLLMs to empower diverse, arbitrary grounding queries. Specifically, we introduce Spatially-Grounded Thought Tokenization to explicitly align special tokens with spatial locations for clear spatial correspondence and visual interpretability. We further design a Context-Adaptive Chain-of-Tokens that dynamically switch grounding modes within an interleaved reasoning chain, achieving robust grounding across tasks of varying complexity. In addition, we construct PR-Bench, a new referring expression comprehension benchmark to evaluate the perception-reasoning gap. Extensive experiments demonstrate that Motto achieves state-of-the-art performance across diverse free-form grounding tasks.