Vision language models controlled to limit location detail in images
Steering Vision-Language Models for Visual-Contextual Privacy in Location Disclosure
Computer Vision and Pattern RecognitionArtificial Intelligence
Summary
Vision-language models can sometimes reveal where a photo was taken by picking up tiny details, which might share too much private information. The authors studied how to make these models only share location details that match the picture’s visible content. They found they could detect the right level of location detail in the model’s internal signals and use that to guide how much information is shared. Their method improves the balance between too much and too little location info without hurting overall model performance.
What this means in practice
- •For photo sharing platforms: Limit location details exposed by vision-language models to protect user privacy based on image content.$Commercial implications: Enables privacy-safe automatic image captioning and content moderation features that protect user location information.
- •For mobile app developers: Control how much location information vision-language models share from user photos to meet privacy standards.
Authors
Wenyue Chen, Yuhan Cheng, Ruixin Yang, Ethan Mendes, Alan Ritter, Jingwei Sun, Hai Li, Yiran Chen
Abstract
Vision-language models (VLMs) can infer sensitive information such as geolocation from subtle visual cues, creating privacy risks when responses disclose more detail than the visual context warrants. We study visual-contextual privacy in geolocation disclosure, where a VLM should reveal location information only at the granularity appropriate to the visual context depicted in the image. We first show that this contextually appropriate granularity is recoverable from pre-generation hidden states, even when generated responses are misaligned, with disclosure levels exhibiting an ordered structure along a common hidden-state direction. Motivated by this, we propose an inference-time route-and-steer framework: an ordinal granularity router maps visual-contextual privacy representations to a disclosure route, and a disclosure-level-conditioned steering module applies route-specific hidden-state interventions during decoding. Across four VLM backbones, our method improves visual-context-aware disclosure alignment, better balances over- and under-disclosure, and incurs only modest changes on general capability benchmarks. It further transfers to another dataset with a distinct data distribution and privacy-risk taxonomy.