Hyperbolic Hierarchical Clustering for Visual Representation Learning
2026-08-24 • Computer Vision and Pattern Recognition
Computer Vision and Pattern RecognitionArtificial Intelligence
AI summaryⓘ
The authors study how vision models combine information from different image parts, a process called token mixing. They found existing methods rely on complex, hard-to-understand techniques like attention and convolutions. To make this clearer, they created ClusterMixer, which mixes tokens by grouping them hierarchically using clustering in hyperbolic space, a way to naturally represent tree-like structures. Their new model, HCFormer, uses ClusterMixer and performs well on tasks like image classification and detection while being more interpretable. They suggest their approach could lead to vision models that are easier to understand.
token mixervision Transformersclusteringhierarchical clusteringhyperbolic spaceimage classificationobject detectioninstance segmentationsemantic segmentation
Authors
Jianan Wei, Guikun Chen, Zhiyuan Weng, Chunchao Guo, Yujia Wang, Wenguan Wang
Abstract
We investigate the token mixer in vision backbones by revisiting clustering, one of the most classic approaches in machine learning. An effective token mixer is a fundamental component of modern vision backbones like vision Transformers, facilitating information exchange between image patches. Mainstream token mixers, which rely on convolution, attention, MLP, or their hybrids, primarily focus on navigating the trade-off between accuracy and computational cost. However, a significant drawback of these methods is their black-box nature; their encoding process is opaque and lacks interpretability. Diverging from these opaque designs, we introduce ClusterMixer, a transparent token mixer that is grounded in a clustering paradigm and interpretable by design. ClusterMixer explicitly formulates the token mixing process through a hierarchical clustering mechanism. To model the natural, tree-like relationships inherent in visual data, the clustering is performed in hyperbolic space, which is well-suited for embedding hierarchies with low distortion. Building on this innovation, we present HCFormer, a new backbone architecture that integrates ClusterMixer with a series of meticulously designed clustering strategies to ensure robust performance across tasks. Extensive experiments demonstrate that HCFormer consistently outperforms its counterparts across diverse tasks, including image classification, object detection, instance segmentation, and semantic segmentation. Considering its transparency and efficacy, we hope HCFormer can facilitate a paradigm shift toward interpretable backbones.