ProtoLIP improves object-level evidence in vision language models
ProtoLIP: From Sentence-Level to Object-Level Evidence Disentanglement
Computer Vision and Pattern RecognitionArtificial Intelligence
Summary
Vision-language models try to connect pictures and text, but they sometimes struggle to focus on just one object when interpreting sentences. The authors introduce ProtoLIP, a method that organizes visual clues into groups linked to words, helping the model pick out evidence from specific objects better. ProtoLIP works without needing extra labeled data or retraining and improves how well models can localize and separate objects in images based on text queries. This approach also makes it easier to understand which parts of the image contribute to the final prediction.
What this means in practice
- •For computer vision engineers: Create vision-language systems that better identify object-specific evidence without extra spatial labels or model retraining.
- •For ai product developers: Build products that explain image-text matches by breaking down exactly which visual features influenced decisions.$Commercial implications: Provides interpretable evidence localization useful for explainable AI features in consumer or enterprise vision-language applications.
Authors
Yan Zhu, Yongbo Chen, Zhengming Ding, Rebecca Faust
Abstract
Query-conditioned vision--language models enable fine-grained interpretation by revealing how visual evidence changes with textual queries. However, evidence conditioned on complete descriptions does not necessarily resolve into object-specific evidence, nor does an exposed evidence map necessarily identify the evidence that constitutes the model's prediction. Across multiple VLM architectures and independent benchmarks, we find that object-level queries often retain evidence from co-occurring objects and shared context. In this paper, we introduce \textbf{ProtoLIP}, a lightweight prototype-mediated evidence layer that organizes reusable visual prototypes into text-derived semantic families and uses query-dependent family routing to constrain which prototypes may provide evidence. Without spatial annotations or backbone retraining, ProtoLIP improves evidence localization and separation across query granularities, with localization gains transferring to independently pretrained VLMs with well-aligned patch--text representations. Despite using only text-derived weak supervision, ProtoLIP remains competitive with a spatially supervised grounding model while maintaining strong matching and competitive image--text retrieval. Crucially, ProtoLIP constructs its matching score directly from localized prototype evidence, enabling the score to be exactly decomposed into semantic-family and prototype contributions.