VISTA: Auditing Semantic Divergence in Vision-Language Models
2026-07-03 • Computer Vision and Pattern Recognition
Computer Vision and Pattern RecognitionArtificial Intelligence
AI summaryⓘ
The authors studied how vision-language models (VLMs) can respond very differently to images with certain visual features like demographics or logos, in ways that text-based checks can't easily detect. They created a tool called VISTA that looks at differences in responses across multiple models to spot unusual or biased behavior. By testing on models with inserted biases, they showed VISTA can successfully identify these issues. When examining several popular models, they found cases where models selectively refused to answer demographic questions, a new kind of behavior not reported before.
vision-language modelsvisual conceptsmodel auditingsemantic entropydistribution divergencefine-tuningbiased datasetsselective refusaldemographic biasblack-box testing
Authors
Junchi Liao, Jiawen Deng, Fuji Ren
Abstract
Vision-language models can exhibit visual concept-conditioned divergence: given images containing demographic features, corporate logos, or ideological symbols, some models produce unusually uniform responses that differ from what peer models say about the same input. These behaviors evade text-only audits because visual concepts cannot be isolated or substituted the way text tokens can. We present VISTA (Visual Inconsistency Screening Through Analysis), a black-box cross-model audit that couples semantic entropy with distribution-based divergence to flag model-specific anomalies. In a controlled study, we implant concept-conditioned stances in three VLMs via fine-tuning on small biased datasets and confirm that VISTA detects them. Auditing six VLMs across 19 topics, VISTA surfaces 142 high-suspicion cases (1.2%) and identifies selective refusal as a previously unreported divergence pattern, where models refuse demographic queries at rates varying from 0 to 65% across groups.