Compositional scene graph helps clip understand complex images better
CS-CLIP: Compositional Scene Graph-guided CLIP for Robust Compositional Reasoning
Computer Vision and Pattern RecognitionArtificial Intelligence
Summary
When computers look at pictures and match them with descriptions, they sometimes struggle to understand how parts of an image relate together, like objects and their relationships. The authors show that previous methods often got confused by specific elements and sometimes performed worse than basic models. To fix this, they created a new method called CS-CLIP that uses detailed scene graphs to find parts in captions and then carefully changes them to teach the model. This helps the model focus more on how the parts combine rather than just surface words, making it better at understanding complex images without needing lots of training examples.
What this means in practice
- •For image search engineers: Improve image search systems by enhancing understanding of complex object relationships using structured negative examples.
- •For autonomous vehicle developers: Increase accuracy in interpreting scenes with complex object interactions for safer environment perception.
Authors
SeongJun Jeong, Minjoon Jung, Woo Suk Choi, Youwon Jang, Byoung-Tak Zhang
Abstract
Vision-language models (VLMs) demonstrate strong performance across compositional reasoning benchmarks, which require reasoning over semantic perturbations of objects, attributes, relations, and their interactions. However, our controlled analysis reveals that existing compositionality-aware VLMs exhibit element-specific biases, often underperforming vanilla CLIP on certain compositional elements. To address this, we propose Compositional Scene Graph-guided CLIP (CS-CLIP), which uses scene graphs to identify compositional elements and construct structured negatives via selective masking. We further retain negatives that are most contradictory to the original caption, forcing the model to rely on compositional structure rather than surface cues. CS-CLIP achieves state-of-the-art compositional reasoning with robust performance across compositional elements. It also preserves general vision-language capabilities such as cross-modal retrieval and downstream visual reasoning, while requiring fewer training samples than prior methods.