Vision language models fail when combining seeing and solving tasks
Seeing and Solving Are Not Enough for Vision-Language Models
Computer Vision and Pattern Recognition
Summary
Sometimes vision-language models can correctly understand a picture or solve a problem on their own, but still give the wrong answer when asked a question that requires both skills together. The authors found that this combined failure happens quite often across different models and data. They introduced a training method called State Realization Tuning that makes the models first explain what they see before answering. This helps fix many of the errors where models fail to combine seeing and solving correctly.
What this means in practice
- •For ai engineers: Improve multimodal AI systems by integrating intermediate output states to reduce combined visual and solving errors in answers.
- •For software developers: Enhance applications that use vision-language models by incorporating output of visual task states before answering questions to boost accuracy.
Authors
Ziheng Wang, Mingxuan Xie, Yilin Liu, Dayan Wu, Yang Li, Pengwen Dai
Abstract
Vision-language models (VLMs) answer visual questions by combining visual information extraction with downstream problem solving. We investigate a fundamental question: Does an incorrect answer necessarily reflect a failure in visual extraction or problem solving? A model may succeed at both abilities when tested separately yet still fail on the original multimodal question, a distinction that overall answer accuracy cannot reveal. To study this, we perform a question-level empirical analysis across multiple VLMs and visual domains. We define an exactly scorable task state (i.e., the visual information sufficient to solve a question) and use it to test whether the same model can extract the required state, solve the question from the ground-truth state, and answer the original multimodal question. We find that composition failures, where extraction and solving both succeed but direct answering fails, account for 17.7% to 75.6% of direct-answering errors across multiple VLMs and datasets. To address this failure mode, we introduce a simple yet effective method, termed State Realization Tuning (SRT). SRT fine-tunes LoRA adapters attached to the language-model layers while keeping the pretrained VLM weights frozen. It trains the model to output the ground-truth task state before the final answer in a single autoregressive response. SRT improves over standard supervised fine-tuning by 1.7 to 14.1 percentage points and repairs 92.5% to 98.1% of diagnosed composition failures. A single LoRA adapter trained with SRT also improves performance across substantially different task-state structures. Our work shows that having both visual extraction and problem-solving capabilities does not guarantee correct multimodal answering. Requiring the model to first output the visual information needed to solve the question can help bridge this gap.