Multimodal models use visual scaffolds to improve reasoning accuracy
Visual Scaffolding: Toward Native Multimodal Reasoning
Computer Vision and Pattern Recognition
Summary
Complex problems are easier to solve when you write down your intermediate steps, like building a frame while constructing a house. The authors found that current AI models struggle to do this with visual information because it's hard to describe images in words alone. They created a method where the AI can directly use and remember visual data as it reasons step by step. Tests show this helps the AI solve tricky tasks better than other popular models. This approach could make AI better at tasks needing both seeing and thinking together.
What this means in practice
- •For multimodal ai developers: Enable AI systems to incorporate and reuse visual data during step-by-step reasoning for improved problem-solving accuracy in complex visual tasks.
- •For robotics software engineers: Use native visual reasoning to better interpret and act on visual environments when robots perform navigation or object manipulation tasks.
Authors
Run Shao, Zhaoyang Zhang, Ziyu Li, Linrui Xu, Ji Qi, Xiu Su, Yichao Cao, Chao Tao, Haifeng Li
Abstract
The success of chain-of-thought (CoT) in complex reasoning can be attributed to a textual scaffolding effect: explicit intermediate results can be examined, reused, and composed to support subsequent reasoning, much as scaffolding supports a building under construction. However, mainstream multimodal LLMs lack an equivalent scaffold for visual distinctions that are difficult to express in concise verbal descriptions. We therefore ask: Can native visual representations become first-class citizens alongside language, providing a visual scaffold for reasoning? We define native multimodal reasoning as reasoning in which native visual representations and language jointly carry reusable intermediate states. As a concrete implementation, the visual scaffolding mechanism we proposed lets the MLLM autonomously initiate a retrieval action at any reasoning step. A lightweight retriever selects native visual features, which are then incorporated directly into the evolving CoT and remain available to subsequent steps. To evaluate this mechanism under controlled visual demands, we introduce NMR-Bench, comprising Burden Counting and Path Tracing. With 88M added parameters (approximately 2%) on Qwen3-VL-4B-Thinking, our model achieves 66.43% and 72.50% accuracy, respectively, surpassing the evaluated GPT-5.5 and Gemini 3.1 Pro baselines. Controlled interventions show that changing selected visual content affects task accuracy and can redirect subsequent visual selection even when intervening text is held fixed. These findings show that native visual states can function as intermediate reasoning material and point toward reusable scaffolds for additional modalities.