Vision language models learn new traits for unusual images
Inductive Visual Logic for Few-Shot Out-Of-Distribution Adaptation in VLMs
Computer Vision and Pattern Recognition
Summary
Generative vision-language models do well on tasks they were trained on but struggle with very different kinds of images, like specialized medical scans. The authors show that even when these models can’t classify such images correctly, they can still describe their visual features accurately. Using this insight, the authors created a method called Inductive Visual Logic (IVL) that builds new classification knowledge from the model’s descriptions without extra training. IVL extracts visual traits from a few examples and uses them to identify classes in tough cases, improving accuracy and providing clear explanations.
What this means in practice
- •For medical imaging teams: Classify rare or specialized medical scans by extracting visual patterns when existing models fail to recognize them directly.
- •For security system developers: Improve recognition of unusual or new threats by building classifiers from descriptive traits using few examples.
Authors
Hung-Jen Chen, Yu-Heng Ho, Ting-Yao Huang, Po-Hsiang Hsu, Li-Yu Chen, Chun-Yi Lee, Min Sun
Abstract
Generative vision-language models (VLMs) such as Qwen-VL and LLaVA achieve strong zero-shot performance on tasks overlapping with their pretraining distribution, yet fail on specialized domains where the required discriminative features were never learned, a regime we term distant out-of-distribution (OOD). Standard adaptation methods cannot overcome this representational absence because they operate within the encoder's existing feature space. However, VLMs retain a robust descriptive capacity even when discrimination collapses: a model that cannot classify a medical scan can still articulate its visual patterns. Exploiting this asymmetry, we introduce Inductive Visual Logic (IVL), a training-free framework that constructs classification knowledge from the model's surviving descriptive ability. IVL extracts visual traits from few-shot support images through dual-mode prompting, combining semantic descriptions with primitive visual observations, and organizes them into per-class trait dictionaries. At inference, hierarchical filtering identifies spatially grounded trait evidence for classification. Across multiple distant-OOD benchmarks, IVL achieves the highest aggregate accuracy under two VLM backbones while producing interpretable, trait-traceable predictions.