Thinking with images achieves better reasoning without coordinates
Thinking with Images without Coordinates
Computer Vision and Pattern Recognition
Summary
Many computer programs that understand images rely on matching exact spots in pictures with words, but this can miss important details and be inefficient. The authors found that using actual picture coordinates has limitations for reasoning about images. They created a new method called TiCO that thinks about images by finding meaningful patterns without needing exact coordinates. TiCO works better and faster on many image understanding tests than previous ways that require coordinates.
What this means in practice
- •For computer vision engineers: Improve image understanding models by enabling reasoning using semantic regions without relying on pixel coordinates.
- •For ai system designers: Build more efficient multi-modal AI systems that perform image-related reasoning tasks faster by avoiding coordinate computations.
Authors
Meng Cao, Haoze Zhao, Yi Wang, Ian Reid, Xiaodan Liang
Abstract
Recent advances in multi-modal large language models have shifted toward the "thinking with images" paradigm to improve reasoning faithfulness, yet existing methods rely heavily on explicit coordinate- based grounding. We identify two fundamental limitations in this approach, i.e., a semantic gap where sparse regressions miss high-dimensional latent structures, and a resolution gap due to the granularity mismatch between patch-level tokens and pixel-level coordinates. To address these, we propose TiCO (Thinking with images without COordinates). It reformulates visual reasoning as a semantic evidence discovery process within the latent space, and evidence is injected as token-level regions without pixel coordinates. Specifically, TiCO utilizes a dedicated probe token to estimate similarity distributions and performs importance sampling over spatially connected components to generate diverse, semantically dominant evidence rollouts. Extensive experiments across multiple high-resolution and general visual reasoning benchmarks demonstrate that our TiCO consistently outperforms state-of-the-art coordinate-based methods with significantly improved inference efficiency.