Hierarchy-aware supervision drives improved image retrieval accuracy
What Drives Hierarchy-Aware Image Retrieval? Taxonomy Alignment, Objective Choice, and Geometry
Computer Vision and Pattern Recognition
Summary
This paper explores why certain image retrieval methods that understand category hierarchies work better. The authors find that training strategies aligned with the actual category tree (taxonomy) matter more than whether the data is embedded in Euclidean or hyperbolic space. They test this using frozen image features and compare different training objectives and embedding geometries. Their results show that carefully aligning the training with the real category structure improves retrieval accuracy more than geometric embedding choices.
What this means in practice
- •For computer vision engineers: Improve image retrieval systems by incorporating taxonomy-aligned objectives to boost strict hierarchical retrieval accuracy.
- •For ecommerce platform developers: Enhance product image search by using hierarchy-aware training objectives that respect category trees for better user relevance.
Authors
Ling Shi
Abstract
Foundation vision models provide strong generic representations, yet high class-level retrieval accuracy does not necessarily imply that an embedding respects a target semantic taxonomy. We study strict explicit-taxonomy image retrieval on frozen DINOv2 features and ask: when hierarchical retrieval improves, how much of the change is associated with the organization of taxonomy-aware supervision, and how much with the Euclidean-hyperbolic geometry choice? We evaluate higher levels with strict cross-class criteria that exclude finer-grained matches, and compare Euclidean and hyperbolic projections trained with taxonomy-distance regression or a taxonomy-aware supervised contrastive objective. A compute-matched 2 x 2 Geometry x Loss factorial uses the same 768-256-32 projector capacity, optimization schedule, batch order, and fixed 100-epoch budget; the Loss axis denotes the Regression-to-Taxonomy-SupCon objective-family contrast. On CUB, the objective-family contrasts in mean hierarchy mAP (strict middle/high average, excluding Class/Leaf) are +0.0487 in Euclidean space and +0.0414 in hyperbolic space, compared with geometry contrasts of +0.0102 and +0.0030. On NABirds Parent-disjoint retrieval, the corresponding objective-family contrasts are +0.0467 and +0.0440, whereas geometry contrasts are +0.0017 and -0.0009. A semantic-alignment control shows that the true taxonomy substantially outperforms a structure-preserving shuffled hierarchy, while a NABirds curvature/radius control does not support stronger negative curvature as the explanation for the observed hierarchy gains. Across the two taxonomies, the Regression-to-Taxonomy-SupCon contrasts are larger in aggregate than the evaluated geometry contrasts; semantic alignment also matters separately, while geometry remains hierarchy-dependent.