Learning Structured Visual Compositional Representations for Weakly Supervised Referring Expression Comprehension
2026-07-06 • Computer Vision and Pattern Recognition
Computer Vision and Pattern Recognition
AI summaryⓘ
The authors study how to identify objects in images based on natural language descriptions when only limited supervision is available. They point out that previous methods use simple object features that don't clearly show relationships between objects. Their new approach, called Structured Visual Compositional Representation (SVCR), explicitly models both individual objects and the relationships between pairs of objects. This helps match image information with language more accurately, and their tests show this method works better than previous ones.
Referring Expression ComprehensionWeakly Supervised LearningVisual RepresentationUnary EmbeddingsRelational EmbeddingsCompositional AlignmentVisual-Textual MatchingRefCOCORefCOCO+RefCOCOg
Authors
Lian Xu, Mohammed Bennamoun, Farid Boussaid, Hamid Laga, Yulan Guo, Dan Xu
Abstract
Referring expression comprehension (REC) aims to localize the object in an image described by natural language. In Weakly supervised REC (WREC), existing approaches primarily operate on anchor-level visual representations. Even when enriched with auxiliary cues, relational interactions remain implicitly encoded within individual anchor features. The resulting visual representation remains flat and unary-only, limiting its ability to align with the structured nature of language. In this work, we propose a Structured Visual Compositional Representation (SVCR) learning framework for WREC. Rather than implicitly encoding relations within unary anchors, the proposed SVCR explicitly models both unary object embeddings and pairwise relational embeddings, forming a structured visual representation space. We further introduce a compositional alignment mechanism that matches unary and pairwise visual representations with their corresponding textual embeddings in a unified manner, enabling compositional visual-textual matching under weak supervision. Extensive experiments on RefCOCO, RefCOCO+, and RefCOCOg show that the proposed SVCR achieves state-of-the-art performance. These results demonstrate the effectiveness of explicit structured visual representations and visual-textual alignment for WREC.