Text steering in visual AI can wrongly profile people by race and crime words
When Pointing Becomes Profiling: Social Bias and Refusal in Text-Steerable Visual Encoders
Computer Vision and Pattern Recognition
Summary
The authors found that a visual AI model called SteerViT, which tries to point at things in pictures based on text prompts, often points at certain people based on racial and social stereotypes instead of what the prompt actually says. For example, when asked to point at a 'criminal,' the model disproportionately pointed at Black people. This bias comes from specific parts of the AI’s design and does not appear in similar models. They created SafeSteer, a method that helps the AI refuse to answer when a prompt might lead to biased or wrong pointing, reducing these social biases while keeping the model’s ability to find things in images.
What this means in practice
- •For computer vision engineers: Improve visual pointing AI systems by adding bias refusal mechanisms to avoid demographic stereotypes when responding to sensitive prompts.
- •For ethics and compliance teams: Evaluate and reduce social biases in AI models used for image analysis to ensure fair and responsible deployment in real-world settings.
Authors
Seyedarmin Azizi, Parsa Razmara, Sajjad Ghiasvand, Ardalan Aryashad, Tina Khezresmaeilzadeh
Abstract
Text-steerable visual encoders condition a frozen Vision Transformer on a natural-language prompt and point at the matching image region, producing prompt-conditioned heatmaps and features. We audit the released SteerViT model and find that its steering interface turns language-driven pointing into demographic profiling. On counterfactual face mosaics and real multi-person photographs, "the person" selects a man more often than the base rate, crime words select Black people more often than occupation words do, "the man" among four women selects the Black woman 75.6% of the time on SocialCounterfactuals, and "the monkey" selects Black people. None of them is in the backbone's unsteered saliency, and the crime association is not in the frozen text encoder; the race effects grow with steering strength and enter through the last two cross-attention layers: giving the prompt to those layers alone reproduces them. Four other grounding models, CLIPSeg, Grounding-DINO, SAM 3, and Qwen2.5-VL, show no crime routing and weaker race routing. The model cannot refuse. Its patch softmax must place its mass somewhere, and its heatmap separates a groundable request ("the police officer") from one that is absent ("the pizza") or that would require a stereotyped demographic inference ("the criminal") only at chance (AUROC 0.52). We introduce SafeSteer, which adds a learned abstention output and counterfactual-invariance training on the same 23M steering parameters. SafeSteer separates should-point from should-abstain prompts at AUROC 0.997, generalizes to held-out sensitive and absent words (0.998; at a 0.5 threshold it refuses 99.8% of trained and 62% of held-out sensitive words) with no over-refusal of visible people on the benchmark, and removes most race routing while keeping localization and retrieval within 1 point of the released model. Code, benchmarks, and the curated mosaic sets will be released.