Visual spikes in large vision language models are brittle and triggerable
Mind the Spike: Mechanisms and Brittleness of Visual Massive Activations in Large Vision-Language Models
Machine LearningArtificial IntelligenceComputation and LanguageComputer Vision and Pattern Recognition
Summary
Some large models that understand both images and text have very large spikes in activity in certain parts of the model, especially when seeing images. These spikes happen in different spots for different images and models, and they can be easily created, moved, or removed by slightly changing the image. The authors discovered how these spikes form and created a way to prevent them without affecting the rest of the image processing. This finding helps understand why these models can sometimes behave unpredictably when images are changed a bit.
What this means in practice
- •For machine learning engineers: Detect and reduce unstable large activations in vision-language models to improve model reliability on varied image inputs.
- •For cybersecurity teams: Design subtle image modifications to identify vulnerabilities or defend against adversarial attacks that exploit activation spikes in vision-language models.
Authors
Jonas Ngnawé, Yann Pequignot, Sabyasachi Sahoo, Christian Gagné, Frédéric Precioso, Sanmi Koyejo
Abstract
Large vision-language models (LVLMs) inherit massive activations from their text-only bases: spikes where a few fixed hidden channels receive values thousands of times above the typical magnitude. The text spike systematically appears in early layers at a fixed initial position, independently of input content. Visual spikes vary across images, but whether their formation follows a consistent pattern across LVLMs and how they respond to image perturbations remain open questions. We find that some LVLMs do not form visual spikes, while others spike at different rates, typically in deeper layers. We identify the trigger direction from model weights and an interpretable location rule: before the language model decoder runs, eventual spike tokens are largely restricted to those sharing least with the rest of the image. Crucially, visual spikes are strikingly brittle. Common corruptions frequently create and relocate spikes, and less often remove them, raising overall incidence. Our trigger-guided spike attack deliberately creates or removes spikes under a small $\ell_\infty$ budget, with 1/255 enough in nine of the ten models that spike. Finally, our preventive intervention removes only the trigger component before spikes erupt, eliminating or substantially reducing spikes on clean and perturbed images while leaving the other image tokens nearly unchanged. Our study spans 25 adapter-based LVLMs built on 18 released text-only bases from 10 families, ranging from 2B to 72B parameters.