Vision language models recognize novel object attribute pairs using geometry
Compositionality with Primitives: A Unified View from Euclidean to Hyperbolic Vision-Language Models
Computer Vision and Pattern Recognition
Summary
Recognizing new combinations of objects and their attributes, like a blue apple, can be hard for computers. The authors show that vision-language models naturally break down images into basic parts, or primitives, which can be combined in simple ways to identify unseen pairs. They found that just using these parts directly works well and that how these parts are combined depends on the shape of the space the model uses, like flat or curved spaces. Their approach works across multiple models and datasets, improving recognition of new object-attribute pairs.
What this means in practice
- •For computer vision engineers: Identify new visual object-attribute combinations without retraining by using primitive embeddings from vision-language models.
- •For machine learning platform teams: Improve model robustness and compositionality by integrating geometric reasoning across Euclidean and hyperbolic embedding spaces.
Authors
Puru Vaish, Meelad Dashti, Christoph Brune, Pascal Mettes, Tatiana Tommasi, Nicola Strisciuglio
Abstract
Compositional zero-shot learning (CZSL) aims to recognize novel combinations of previously observed objects and attributes. Prior work showed that large-scale vision-language models encode decomposable semantic structures that can be recomposed to enable generalization. The Geodesically Decomposable Embeddings (GDE) framework formalizes this by identifying directions in the tangent space at the Frechet mean of the embedding distribution, which encode primitive visual concepts. Unseen attribute-object compositions can be synthesized through vector addition of these primitives. We revisit the assumption that compositional generalization requires explicit attribute-object pair re-composition. By means of new compositional scoring functions and extensive evaluations, we show that directly using the decomposed primitives suffices for recognizing unseen compositions. In that, we gain insights on how the geometry of the embedding manifold (Euclidean, spherical, or hyperbolic) and dataset structure affect compositional reasoning. Across models (CLIP, DINOv2, MERU, HyCoCLIP), we show that compositional structure consistently emerges as primitive representations, enabling inference on CZSL benchmarks (MIT-States, VAW-CZSL, UT-Zappos50K, Clothing16K) and group robustness on Waterbirds and CelebA. Code: https://github.com/nis-research/Compositionality-with-Primitives.