Vision language models show varied sensitivity to image and text changes
Search-Based Testing of Modality Sensitivity in Vision Language Models
Computer Vision and Pattern RecognitionSoftware Engineering
Summary
Vision-language models understand pictures and words together, but testing often changes pictures or words separately. The authors created a new way to test these models by changing images and text at the same time in small ways. This method finds that some models are more sensitive to changes in words, others to images, and some to both together. People can still understand the changed images and text better than the models can. This shows that testing both inputs together reveals weaknesses that separate tests might miss.
What this means in practice
- •For machine learning engineers: Identify specific weaknesses in vision-language models by testing combined small changes to images and text inputs during development.
- •For software testers: Evaluate and improve robustness of vision-language-based applications by leveraging joint perturbation testing to expose errors missed by separate image or text tests.
Authors
Oliver Weißl, Eduardo Breitembach, Andrea Stocco
Abstract
Vision-language models (VLMs) process images and text jointly, yet robustness evaluations often perturb these inputs separately or assess only predefined corruption levels. Such evaluations provide limited evidence of which modality is more susceptible to perturbations and whether smaller changes across both modalities can jointly expose failures. We present MoST, a black-box testing framework that characterizes modality sensitivity by comparing image-only, text-only, and joint perturbations. MoST uses multi-objective search to select and combine perturbations and adjust their severity, seeking prediction degradation while limiting changes to images and prompts. We evaluate MoST on four VLMs performing object localization on ImageNet and driving images. The models exhibit distinct sensitivities: textual perturbations dominate for some, whereas others are susceptible across both modalities. In several configurations, joint perturbations match or exceed the effectiveness of single-modality perturbations with smaller modifications to each input. In a human study, 60-76% of assessments across models judge the perturbed inputs sufficiently understandable to perform the task, and human localization is more accurate than that of the corresponding VLMs. These results show that evaluating modalities separately can overlook failure-inducing combinations and that modality sensitivity provides a more detailed account of VLM robustness than aggregate failure rates alone.