Visual inputs shift attribute descriptions in vision language models
Visual Input and Its Framing Affect Attribute-based Descriptions Produced by Large Vision-Language Models
Computer Vision and Pattern Recognition
Summary
The researchers found that when large vision-language models (LVLMs) are given both an image and a text prompt, the image changes the model’s response, even if the prompt doesn’t refer to the specific image shown. For example, asking about a dog breed’s features yields different answers depending on the dog pictured and how it is shown. The way the dog is framed in the image influences whether the description focuses more on physical traits or situational context. This shows that visual context plays a bigger role than expected in how these models generate descriptions.
What this means in practice
- •For ai product developers: Improve product design by considering how embedded images influence LVLM attribute descriptions.
- •For ux designers: Design interfaces that carefully frame images to guide LVLM-generated descriptions for better user experience.
Authors
Xiaomeng Wang, Martha Larson, Zhengyu Zhao
Abstract
Large vision-language models (LVLMs) are commonly used with only a single text prompt as the input, or plus an image. In this paper, we demonstrate that when the image exists, even if the text prompt is not about the specific instance (but only the concept it belongs to) in that image, the response would still be affected. For example, when the text prompt only asks for the attribute descriptions of a dog breed, an image depicting a specific dog from that breed would shift the response. Further, how the specific instance is framed in that image would determine towards which the response shifts. Detailed analyses also reveal that in the response, physical terms increase from 18% for text-only to 45% (40%) for subject-focused (subject-in-situation) framings. Overall, the unexpected effects of visual cues on LVLMs highlight the need to understand the presence of an image and its framing when evaluating the robustness of LVLMs.