Vision language models learn new visual names from context and images
Seeing Before Naming: Context-Grounded Learning of Novel Visual Concepts
Computer Vision and Pattern Recognition
Summary
Identifying a new kind of object or image category is not just about recognizing its shape or features, but also about linking it to a meaningful name. The authors propose a method called SynerNet that helps models learn new concepts by combining visual clues with appropriate language context. This approach improves how well models can name unfamiliar objects after only a few examples. They tested SynerNet on various datasets and showed that it works better than previous methods that only learned names or visuals alone.
What this means in practice
- •For computer vision engineers: Improve few-shot classification systems by integrating contextual language information to better recognize and name new visual categories.
- •For natural language processing teams: Enhance language grounding in multimodal models by conditioning textual concept representations on visual examples for better semantic understanding.
- •For augmented reality developers: Develop AR applications that can identify and name novel objects in real time using visual and linguistic context together.$Commercial implications: Enables AR interfaces to dynamically learn and label new objects, improving user experience and product differentiation.
Authors
Gabriel Adineera, Jinoh Kim
Abstract
Learning an unfamiliar visual category requires more than adapting a classifier: the model must connect its visual evidence to a useful semantic name. A pre-trained vision-language model can retain discriminative image features while providing a weak textual anchor for the target concept. We address this imbalance through context-grounded semantic anchoring. Our framework, SynerNet, learns concept-specific name embeddings, exposes them to compatible linguistic contexts, and conditions their representations on visual evidence. Difficulty-aware processing and adaptive objective weighting coordinate this adaptation while retaining the pre-trained backbone. On four VISTA-Beyond domains, SynerNet reaches a mean 16-shot accuracy of 74.35%, compared with 72.45% for few-shot name learning. Across 14 domains in the name-masked transfer setting, it improves mean accuracy from 45.17% to 46.74%. Ablations identify concept-specific name embeddings and cross-modal context integration as important components. These results support treating concept acquisition as the construction of visually grounded semantic anchors.