Vision language models reveal layer specific safety control points
Depth-Dependent Safety Geometry in Vision-Language Models: Layer-Wise Localization and Bidirectional Refusal Control
Computer Vision and Pattern RecognitionComputation and LanguageCryptography and SecurityMachine Learning
Summary
Vision-language models combine images and text to understand content, but it’s unclear how they decide to refuse harmful inputs. This paper studies how different layers inside these models handle safety and shows that early layers can strongly control refusal decisions by adjusting activations. Later layers separate safe from unsafe cases but are less able to steer refusal directly. The authors also uncover a potential way to trick these refusal systems during inference. Their findings help make the safety mechanisms in frozen vision-language models clearer and suggest ways to improve them without retraining.
What this means in practice
- •For ai safety engineers: Identify and strengthen safety-critical layers in vision-language models to improve refusal reliability without retraining.
- •For security analysts: Detect and evaluate possible attacks that bypass safety refusals in vision encoders by exploiting layer-specific activation patterns.
Authors
Muhammad Zeeshan Karamat, Syed Talal Hassan, Christiana Chamon Garcia
Abstract
Vision-Language Models (VLMs) are widely deployed, yet the internal mechanisms behind their safety behavior remain poorly understood. We present a lightweight inference-time framework for localizing and manipulating safety-sensitive representations in frozen VLMs, without retraining or modifying parameters. On Qwen3-VL-4B, layer-wise activation divergence analysis reveals non-uniform safety geometry across depth. Causal interventions show that early vision layers support strong bidirectional refusal control through activation steering along a learned safety direction, with unsafe rejection spanning 6%-96%, whereas later layers respond more to activation magnitude scaling and exhibit a substantial safety-selectivity trade-off. Deeper layers remain well separated along the safety direction yet are largely insensitive to steering, distinguishing representational separability from causal controllability. InternVL2.5-4B shows qualitatively similar depth-dependent activation geometry despite differences in layer location and magnitude. Our results expose a white-box, representation-level refusal-bypass attack surface in VLM vision encoders and motivate inference-time stabilization of safety-sensitive representations.