HyFL-CLIP: Hyperbolic Fine-Tuning of CLIP for Robust Long-Context Understanding
2026-07-01 • Computer Vision and Pattern Recognition
Computer Vision and Pattern Recognition
AI summaryⓘ
The authors explain that the current CLIP model has trouble understanding long text descriptions because it treats the text as a simple list and doesn't handle complex structures well. They introduce HyFL-CLIP, a new way to train CLIP that uses hyperbolic geometry to better capture relationships within long texts and between text and images. This method helps the model understand parts of the text and how they relate to the whole, improving its accuracy especially when text is changed or reordered. Their experiments show that HyFL-CLIP performs better than previous methods on tasks involving long and complex text alongside images.
CLIPContrastive LearningHyperbolic SpaceCross-modal RetrievalPositional EncodingHierarchical SemanticsText-Image AlignmentEinstein MidpointStable Diffusion XLTextual Perturbations
Authors
Ji Ha Jang, Hayeon Kim, Chulwon Lee, Junghun James Kim, Se Young Chun
Abstract
CLIP (Contrastive Language-Image Pre-training) has become a de facto paradigm for image-text alignment, but it struggles with long-context descriptions (>77 tokens) due to absolute positional encoding and pretraining on short captions. In long contexts, sentences are often reordered, summarized, or partially omitted. Although prior works extend CLIP with longer positional encodings, they often suffer from degraded image-text alignment under such text perturbations. We attribute this limitation to the Euclidean contrastive objective, which enforces strict one-to-one matching and lacks explicit mechanisms for modeling hierarchical relationships between global context and its constituent elements. To address this issue, we propose HyFL-CLIP, a hyperbolic fine-tuning framework that distills the well-established text-image alignment learned in Euclidean CLIP into hyperbolic space via cross-manifold similarity distillation, leveraging its geometry to capture hierarchical and entailment relations. Our method models hierarchical semantics by linking summarized token-wise features, long-context descriptions, constituent short textual components, and images, capturing part-whole relationships via hyperbolic entailment with Einstein midpoint aggregation. Experiments on diverse benchmarks, including long-context cross-modal retrieval, cross-modal retrieval with caption perturbations, intra-modality retrieval, and short-text cross-modal retrieval, show that HyFL-CLIP achieves more robust long-context understanding. In particular, it yields up to 19.5% improvement in long-text cross-modal retrieval under textual perturbations over the best prior method. We also show HyFL-CLIP can be seamlessly integrated into other model frameworks by applying it to Stable Diffusion XL (SDXL).