Scene graphs improve reasoning in vision language models
CS-CLIP: Compositional Scene Graph-guided CLIP for Robust Compositional Reasoning
Computer Vision and Pattern RecognitionArtificial Intelligence
Summary
Vision-language models try to understand pictures and text together, especially when comparing complex scenes that involve different objects and their relationships. The authors found that many of these models struggle with certain detailed parts of a scene, sometimes doing worse than simpler models. They created CS-CLIP, which uses scene graphs to highlight important elements and trains the model to focus on these by creating tricky examples to learn from. This method helps the model better understand complex scenes and still works well on other tasks, needing less training data.
vision-language modelscompositional reasoningscene graphsCLIPstructured negativessemantic perturbationscross-modal retrievalvisual reasoning
Authors
SeongJun Jeong, Minjoon Jung, Woo Suk Choi, Youwon Jang, Byoung-Tak Zhang
Abstract
Vision-language models (VLMs) demonstrate strong performance across compositional reasoning benchmarks, which require reasoning over semantic perturbations of objects, attributes, relations, and their interactions. However, our controlled analysis reveals that existing compositionality-aware VLMs exhibit element-specific biases, often underperforming vanilla CLIP on certain compositional elements. To address this, we propose Compositional Scene Graph-guided CLIP (CS-CLIP), which uses scene graphs to identify compositional elements and construct structured negatives via selective masking. We further retain negatives that are most contradictory to the original caption, forcing the model to rely on compositional structure rather than surface cues. CS-CLIP achieves state-of-the-art compositional reasoning with robust performance across compositional elements. It also preserves general vision-language capabilities such as cross-modal retrieval and downstream visual reasoning, while requiring fewer training samples than prior methods.