SteerProbe reveals limits of safety steering in vision language models
SteerProbe: Learning to Bypass Safety Steering in Vision-Language Models
Cryptography and Security
Summary
Vision-language models aim to avoid harmful outputs by steering their internal workings during inference without changing their core. The authors find that simple rewordings or changes in images can trick this safety steering, causing the models to produce harmful content despite protections. They introduce SteerProbe, which learns to create such trick inputs by probing these models without needing to see inside them. This shows current safety measures are not enough, and future protections should handle different ways harmful intent can appear.
What this means in practice
- •For ai safety engineers: Test safety steering defenses for vision-language models against input rewordings and visual variations using output-only black-box attacks.
- •For ai security teams: Improve robustness evaluation of multimodal safety tools by including tests for intent-preserving input changes that bypass steering corrections.
Authors
Xinwei Zhang, Aoting Hu, Hangcheng Liu, Shuchao Pang, Qingqing Ye, Haibo Hu
Abstract
Activation steering offers an inference-time defense for vision--language models (VLMs) by modifying intermediate representations without updating backbone parameters. However, protection on benchmark inputs may not persist across alternative expressions of the same harmful request. We investigate this gap using fixed textual, visual, and joint reformulations designed to preserve the underlying intent, and find that these changes can bypass representative steering defenses. A complementary local analysis provides a sufficient condition under which a reformulation can cross a surrogate safety margin despite any admissible change in the local steering correction. We then introduce SteerProbe, an output-only black-box attack that learns to select effective reformulations for unseen requests from a shared calibration budget. Across three VLM backbones, two benchmarks, and three steering defenses, SteerProbe increases Harmful Rate in all defended settings using 500 total calibration queries per endpoint and benchmark, raising the average from 7.43% to 18.36%. These findings highlight that robustness on original benchmark inputs is insufficient to characterize the safety of steering defenses and motivate reformulation robustness as an important evaluation dimension. They further motivate steering mechanisms that preserve safety across intent-preserving multimodal variations while maintaining benign utility.