Hierarchical prompts improve hyperbolic vision language models for classification
Hierarchical Prompt Learning for Hyperbolic Vision-Language Models
Computer Vision and Pattern Recognition
Summary
Vision-language models that understand images and text together often arrange information in a hierarchy, like a family tree. However, most methods treat categories as unrelated flat lists. The authors propose a way to teach models using the existing hierarchy, helping them learn parent and child class relationships better. This approach improves how well models identify new categories and transfer knowledge between datasets while keeping predictions consistent with known taxonomies.
What this means in practice
- •For machine learning engineers: Use hierarchical prompt learning to improve classification accuracy for models deployed on new or evolving image classes within existing taxonomies.
- •For computer vision developers: Enhance domain adaptation capabilities of vision-language models when transferring to datasets with hierarchical label structures.
Authors
Andro Erdelez, Pascal Mettes, Behzad Bozorgtabar
Abstract
Hyperbolic vision-language models (VLMs) represent image and text features in a geometry naturally suited to hierarchy, but their adaptation to downstream tasks has largely relied on fixed prompts. Existing prompt learning methods, meanwhile, treat class labels as a flat set and do not exploit available taxonomic structure. We address this gap with a hierarchical prompt learning plug-in for frozen hyperbolic VLMs. Given a fixed offline parent-class hierarchy, it augments a class prompt learner with a separate parent prompt learner, parent-level supervision, hyperbolic entailment regularization, and parent-feedback logit fusion. We instantiate the method with CoOp, CoCoOp and MaPLe, yielding HyPLO, CoHyPLO and MaHyPLO. Across the standard 11-dataset benchmark, all variants improve base-to-new generalization and cross-dataset transfer, and remain comparable to their prompt learning baselines under domain shift. Six hierarchical metrics and embedding analyses show that the method produces more taxonomically consistent predictions and induces a hierarchy-consistent organization of parent, class, and image embeddings in hyperbolic space. Its gains are largest when novel classes must be placed within a fixed taxonomy, and smallest for fine-grained confusions among sibling classes or shifts affecting only the image distribution.