Behavioral affinity boosts zero-shot vision-language classification accuracy
Behavioral Affinity Transduction for Zero-Shot Vision-Language Classification
Computer Vision and Pattern Recognition
Summary
Sometimes it’s hard for AI systems to correctly identify what's in a picture without any examples to learn from. The authors found a way to improve this by having the system look at groups of pictures together instead of one at a time. They connect images based on how the AI rates possible labels, forming a network that helps guess better on each image. This method, called Behavioral Affinity Transduction, improved performance on different datasets without needing any labeled examples.
What this means in practice
- •For computer vision engineers: Improve image recognition accuracy when no labeled training examples are available by using group relationships between unlabeled images.
- •For automated surveillance teams: Enhance object detection in batches of unlabeled images by using learned affinities to better categorize difficult-to-classify subjects.
Authors
Junxi Huang, Wenshuo Chen, Xiucheng Wang, Meitong Shen, Shaofeng Liang, Yuehan Kou, Yutao Yue, Nan Cheng
Abstract
An image that is hard to classify on its own may have a helpful neighbor in the same unlabeled batch. Zero-shot vision-language models (VLMs) score images independently and leave this evidence unused. We study small unlabeled query batches with no labeled examples. Visual similarity can reflect pose or background, so it may connect images from different classes. Each image's scores over the class vocabulary offer another way to choose neighbors. We introduce Behavioral Affinity Transduction (BAT), which builds a sparse graph from these class-response profiles. The graph supplies neighbor evidence during prediction. For EM-Dirichlet, it also screens candidate classes and initializes assignments. BAT plugs into direct and iterative readouts of a frozen VLM. Across diverse datasets, BAT improves the mean accuracy of every tested readout. In the main RN50 comparison, soft and hard EM-Dirichlet gain $4.22$ and $2.72$ percentage points.