Multimodal attacks find weak spots in language AI understanding images
Where Can We Fool the Model's Mind? Response-Guided Semantic Probing and Adaptive Routing for Transferable Attacks on Black-Box MLLMs
Computer Vision and Pattern Recognition
Summary
Some AI models combine language and images to understand the world, but they can be tricked by small changes in pictures. The authors created a method called VISTA that finds exactly where these AI models are most likely to fail by watching how the models answer questions about images. This method works without needing to look inside the models and can fool even commercial AI systems. Their approach can both guess weak points using open AI models and also improve attacks by asking the target model questions.
What this means in practice
- •For ai security teams: Identify and guard vulnerable visual regions in multimodal AI models to strengthen defenses against image-based attacks.
- •For multimodal ai developers: Design and test robust multimodal models by probing semantic weaknesses without requiring internal model access.
- •For content moderation teams: Detect manipulated images that exploit visual-language model vulnerabilities to prevent misuse in commercial platforms.$Commercial implications: Enables tools that automatically flag or block adversarially altered images targeting AI moderators, protecting platform integrity.
Authors
Ahmed Abdelnaby, Mohamed Elmahallawy
Abstract
Multimodal Large Language Models (MLLMs) ground language in visual evidence, yet their semantic reasoning can be exploited through transferable black-box adversarial attacks. Existing attacks typically allocate perturbations using surrogate-model saliency, which may not reflect how an unseen MLLM semantically interprets or relies on different image regions. We introduce VISTA (Vulnerability-Informed Semantic Transfer Attacks), a black-box framework that identifies semantically vulnerable visual regions from an MLLM's responses without access to gradients, logits, parameters, or internal representations. VISTA supports two attack settings: (i) VISTA-Proxy, a pure-transfer setting that probes an open-source MLLM and transfers the resulting adversarial examples to unseen commercial MLLMs without querying them; and (ii) VISTA-Victim, a query-assisted setting that probes the target MLLM to capture model-specific semantic vulnerabilities. VISTA builds an Uncertainty-Based Multimodal Behavior (UBMB) map from response drift, conflict, misleading, and semantic shifts, then uses entropy-adaptive Sinkhorn routing and a surrogate ensemble to focus perturbations on vulnerable regions. Across GPT-5, Gemini-3.5-Flash, and Claude Sonnet 4.6, VISTA achieves up to 97 % attack success rate (ASR) while maintaining a Peak Signal-to-Noise Ratio (PSNR) above 30 dB. Notably, VISTA-Proxy achieves up to 93 % ASR without querying the final victim, demonstrating that response-derived semantic vulnerability can enable effective transfer across heterogeneous MLLMs.