Fully hyperbolic vision transformer improves image recognition accuracy
Rethinking the Fully Hyperbolic Vision Transformer in Polar Coordinates
Computer Vision and Pattern Recognition
Summary
Hyperbolic space helps organize data with natural hierarchies, but previous methods using it ran into numerical problems that limited their effectiveness. The authors rethought how to build transformer models using hyperbolic geometry by working in polar coordinates, which reduces calculation errors. They also introduced new ways to handle positional information and combine features in this space. Their updated transformer model performs better than existing Euclidean and hyperbolic versions on common vision tasks and works well across different datasets.
What this means in practice
- •For computer vision engineers: Improve image recognition systems by using hyperbolic transformers that reduce numerical errors and better capture hierarchical features.
- •For machine learning platform developers: Integrate polar-coordinate-based hyperbolic transformer blocks to build models that generalize well across various image datasets.
Authors
Ahmad Bdeir, Niels Landwehr
Abstract
Hyperbolic space can embed intrinsic hierarchies in data with low distortion due to the exponential growth of volume with distance from the origin. However, current Lorentz transformer blocks are formulated in ambient coordinates, where numerical errors increase at large radii due to instability in the Lorentzian inner product, leading many models to limit the radius to avoid this issue. This prevents us from utilizing the regions of hyperbolic space that motivate the geometry. To address this, we revisit the components of the transformer block in polar coordinates, where hyperbolic operations such as distance calculation and attention centroids can be computed without the numerical cancellation in their ambient-coordinate formulations. Specifically, we propose a polar fully connected layer that separately maps an embedding's direction and radius, allowing the radius of a feature to be learned rather than determined by the norm of a linear map. We additionally introduce horospherical shifts as relative positional encodings whose query-key distances grow logarithmically with the token gap. Finally, we reformulate the residual connection as average radius Lorentz boosts. Combining these components, we develop a fully hyperbolic transformer that substantially improves performance over Euclidean and hyperbolic baselines on standard vision tasks. We further evaluate our model on ImageNet and demonstrate its ability to generalize to other datasets using pre-trained weights, similar to Euclidean counterparts.