ReCoG improves personalized concept recognition using relational comparisons
ReCoG: Relational Concept Graph for Training-Free Personalization
Computer Vision and Pattern Recognition
Summary
Recognizing user-defined visual concepts from only a few examples is hard, especially when the concepts look very similar. The authors show that comparing concepts directly to each other, instead of individually, helps the model distinguish between confusing categories. They propose ReCoG, a method that builds a graph of concepts and the differences between them to improve recognition without retraining. Tests show that ReCoG works better than previous methods, especially for fine-grained and ambiguous concepts.
What this means in practice
- •For mobile app developers: Add personalized object recognition to apps by comparing user-defined concepts directly to improve identification accuracy without model retraining.
- •For robotic vision teams: Enhance robots’ ability to distinguish fine-grained or similar objects by leveraging relational comparisons of concepts during recognition.
Authors
Changseon Yu, Hyeokjun Kweon
Abstract
Training-free personalization adapts a frozen vision-language model (VLM) to recognize user-defined concepts from only a few exemplars by retrieving concept entries at inference time. While recent retriever-based methods enrich each concept with descriptive evidence, they largely treat concepts in isolation, so the retrieved information can be plausible yet non-discriminative, especially when the personalized concept set contains highly confusable, fine-grained categories. We argue that personalization is inherently relational: correct recognition depends on the specific distinctions between a concept and its nearest alternatives. To this end, we propose ReCoG, a Relational Concept Graph for retrieval-augmented personalization. ReCoG represents concepts as nodes and augments them with directed edges that explicitly capture discriminative cues between concept pairs, elicited by a frozen VLM during database construction. At inference, ReCoG retrieves a shortlist and resolves ambiguity via explicit pairwise comparisons conditioned on the corresponding relational edges, selecting the concept that consistently wins against competing candidates. We also introduce FINGER benchmark to test personalized recognition on confusable concept sets. Across FINGER, ReCoG substantially outperforms prior methods, with the largest gains in the most confusable regimes. ReCoG also achieves consistently best performance on established personalization benchmarks including captioning and VQA, confirming that relational evidence is broadly beneficial beyond fine-grained settings. Project Page: https://csy-98.github.io/ReCoG/