Hyper3-CLIP: Hierarchy-Conditioned Hyperbolic Vision-Language Training
Computer Vision and Pattern RecognitionArtificial Intelligence
Summary
The authors developed Hyper3-CLIP, a new vision-language model that better understands relationships within images, such as parts of objects and their connections, by combining hyperbolic geometry with query-based visual pooling. They use different levels of text queries, from full captions to small phrases, to guide the model in learning these relationships during training. Their approach improves image-text retrieval and multi-label classification performance on standard datasets while maintaining good hierarchical understanding. The model activates special pooling only when training, helping it learn detailed visual structures more effectively.
Authors
Matin Mahmood, Antonio Rueda-Toicen, Mohamed ElBassat, Seifeldin Elkerdany, Weixing Wang, Gerard de Melo
Abstract
CLIP-like vision-language models (VLMs) trained with contrastive objectives learn strong global image-text representations, but their Euclidean embeddings and global pooling fail to encode relational structure such as part-whole and parent-child relations. Hyperbolic VLMs address this gap with entailment-based objectives, and text-conditioned variants improve fine-grained alignment through sentence- and phrase-level queries. However, these two lines of work remain separate: hyperbolic VLMs use static image and region features, while query-conditioned methods lack hierarchical geometric structure. We present Hyper3-CLIP, a hierarchy-conditioned hyperbolic VLM that combines global, local, and global-local contrastive learning with query-conditioned visual pooling. To train the model, we construct lightweight query hierarchies from text, comprising full captions, sentence fragments, localized part descriptions, and extracted phrases. Each query conditions the pooling of visual patches, and the resulting representations support image-text, whole-part, and parent-child entailment losses. Query-conditioned pooling is active only during training. Hyper3-CLIP improves R@5 and R@10 retrieval on COCO and Flickr, as well as multi-label classification on VOC and COCO, while remaining competitive on hierarchy metrics. We also audit zero-shot prompt sensitivity under fixed prompt regimes and study the effect of the localized GRIT part budget used during training. Code is available at https://github.com/Hyper3Labs/hyper3-clip.