Vision language models do not see as described personas would see
Telling Is Not Seeing: Textual Personas Rarely Make Vision-Language Models Answer Like the People They Describe
Computer Vision and Pattern RecognitionArtificial Intelligence
Summary
The paper finds that vision-language models (VLMs) prompted to adopt a persona, such as a person with low vision, do not actually perceive images as the described person would. Instead, these models answer based on their knowledge about these conditions rather than experiencing or simulating the visual changes that come with them. The authors tested many models and found that simply describing a blur or color blindness in the prompt did not change model recognition, whereas physically altering an image did. Thus, giving a model a persona makes it say what it knows, not see what the persona would see.
What this means in practice
- •For user experience researchers: Avoid using textual personas with VLMs to simulate user visual impairments as they do not affect the models' visual perception or answers.
- •For vision-based ai developers: Design vision models that incorporate actual image modifications rather than textual description to simulate impaired vision conditions effectively.
Authors
Jiawei Chen
Abstract
Vision-language models (VLMs) are increasingly prompted with personas, such as "You have low vision", to stand in for people in user studies. We ask whether a persona makes a model see like the person it describes. We compare persona answers with measured people with low vision, color blindness or hearing loss, and with simple rules that use only the facts the persona states. The persona never beats these rules, and it erases the real differences between groups and individuals. Describing blur leaves recognition unchanged in all 14 models we test, while blurring the photo lowers it as it does for people. Inside open models, the description changes neither the image information nor the answer probabilities. Yet the models know the effects they fail to show: they can estimate how blur and illusions affect people. Telling is not seeing: a persona makes a model say what it knows about people, not see what they see.