Papers for

vision-based ai developers

Papers whose findings have a practical use for this group, as judged from the abstract. Open a paper to read what it means in practice.

Vision language models do not see as described personas would see

Telling Is Not Seeing: Textual Personas Rarely Make Vision-Language Models Answer Like the People They Describe

Abstract: Vision-language models (VLMs) are increasingly prompted with personas, such as "You have low vision", to stand in for people in user studies. We ask whether a persona makes a model see like the person it describes. We compare persona answers with measured people with low vision, color blindness or hearing loss, and with simple rules that use only the facts the persona states. The persona never beats these rules, and it erases the real differences between groups and individuals. Describing blur leaves recognition unchanged in all 14 models we test, while blurring the photo lowers it as it does for people. Inside open models, the description changes neither the image information nor the answer probabilities. Yet the models know the effects they fail to show: they can estimate how blur and illusions affect people. Telling is not seeing: a persona makes a model say what it knows about people, not see what they see.

Thu 29 OctComputer Vision and Pattern RecognitionArtificial Intelligence
The gist
The paper finds that vision-language models (VLMs) prompted to adopt a persona, such as a person with low vision, do not actually perceive images as the described person would. Instead, these models answer based on their knowledge about these conditions rather than experiencing or simulating the visual changes that come with them. The authors tested many models and found that simply describing a blur or color blindness in the prompt did not change model recognition, whereas physically altering an image did. Thus, giving a model a persona makes it say what it knows, not see what the persona would see.
Open → 2610.39331v1