Vision based method reduces gender bias in language AI models
ViD: Vision-Dominant Gender Bias Mitigation for Large Vision-Language Models
Computer Vision and Pattern RecognitionMultimedia
Summary
Large AI systems that combine images and text can sometimes make unfair guesses about gender. The authors show that by carefully looking at how the AI pays attention to pictures versus words, they can reduce these gender biases. They introduced a technique called ViD that uses new ways to balance visual and language clues without needing more training. This makes the AI's answers fairer and better linked to what it actually sees in pictures. Their tests show that ViD improves results on standard benchmarks without extra cost.
What this means in practice
- •For ai product developers: Integrate ViD to reduce gender bias in AI systems that generate captions or descriptions from images.$Commercial implications: ViD enables fairer AI image captioning products and services, improving trust and market acceptance.
- •For software engineers: Improve fairness and reliability in large vision-language models used in applications without retraining or extra compute during deployment.
Authors
Zhipeng Zhao, Zhaoqiang Wei, Peishun Liu, Youwei Zhao, Ruichun Tang
Abstract
Gender bias in large vision-language models (LVLMs) undermines their fairness and reliability, compromising output trustworthiness. Current mitigation methods rely on training-phase adjustments or post-hoc calibration, but face limitations in dynamic visual bias mitigation. These include inability to capture real-time visual-textual incongruence, dependence on predefined gender bias taxonomies, and degraded cross-modal alignment with emergent bias patterns. To address these challenges, we propose ViD, a causally-inspired framework that analyzes attention mechanisms across five distinct patterns, revealing confounding effects from strong language priors. ViD demonstrates that visual-to-language cross-attention effectively suppresses bias while preserving general reasoning capabilities and text generation quality. ViD incorporates dual mechanisms: backdoor adjustment counters strong language priors, while refined token selection in decoding layers optimizes processing. This enhances model robustness and inference efficiency. Our integrated approach significantly mitigates gender bias across multidimensional social attributes in LVLMs, improving visual grounding and output fairness. Cross-benchmark validation shows ViD reduces gender bias by 14.7\% on single-attribute evaluations (FACET) and achieves significant improvements on image captioning tasks (MS COCO), with gender bias score improving from 0.6708 to 0.9978 for LLaVA. Crucially, these improvements require no additional training overhead, making ViD a scalable and practical solution for bias mitigation in LVLMs.