Learning Taxonomic Trees with Hierarchical Representation Regularization for Large Multimodal Models
2026-07-03 • Computer Vision and Pattern Recognition
Computer Vision and Pattern RecognitionArtificial Intelligence
AI summaryⓘ
The authors found that large multimodal models, which understand both images and text, often struggle to recognize categories in a clear hierarchy (like knowing that a dog is a type of animal). They created a method called HiR2 that helps these models learn better hierarchical relationships by combining visual features and language cues. This method uses special math to keep the hierarchy intact while also making similar things distinct enough from each other. Their tests showed that HiR2 improves the model's understanding of category hierarchies without disrupting other features.
taxonomyhierarchical visual recognitionlarge multimodal modelssemantic relationshipsregularizationLorentz modelentailment conesangular separationembedding space
Authors
Hulingxiao He, Zhi Tan, Yuxin Peng
Abstract
Taxonomies provide key information about the semantic relationships between concepts and the inherent organization of vision and language. Despite their impressive capabilities, large multimodal models (LMMs) often lack taxonomic knowledge, leading to low hierarchical visual recognition (HVR) consistency. These models typically only rely on language modeling objectives during fine-tuning and lack explicit taxonomy-aware regularization. To address this, we propose Hierarchical Representation Regularization ($HiR^2$), a simple plug-and-play regularizer that improves hierarchical consistency in LMMs. Specifically, we introduce a semantic-aware visual tree construction framework that extracts coarse-to-fine visual features from intermediate LLM layers guided by textual cues. The regularizer combines two complementary objectives: a taxonomic entailment loss that enforces hierarchy via hyperbolic entailment cones in the Lorentz model, and a discriminative dispersive loss that promotes angular separation of semantically similar embeddings on the unit sphere without disturbing the radial hierarchical structure. Extensive experiments demonstrate that $HiR^2$ effectively captures taxonomic structures across diverse LMMs and fine-tuning methods. Code is available at https://github.com/PKU-ICST-MIPL/HiR2_ICML2026.