Geometry aware hyperbolic quantization improves hierarchical data representation
Geometry-Aware Hyperbolic Residual Quantization
Machine LearningArtificial Intelligence
Summary
Turning continuous data into discrete tokens is important for many technologies, but most methods use flat (Euclidean) geometry even when the data has a natural hierarchy. The authors show that using hyperbolic geometry, which is better suited for hierarchies, is tricky due to mathematical inconsistencies. They fixed these by introducing new ways to combine and train hyperbolic codes, resulting in more stable and organized token representations. Their method works better than simple hyperbolic adaptations, especially when the data has a hierarchical structure, though for pure compression traditional methods still perform better.
What this means in practice
- •For machine learning engineers: Improve hierarchical representations in tasks like prediction and recommendation using geometry-aware hyperbolic quantization.
- •For computer vision developers: Create more stable image tokenization methods by applying hyperbolic residual quantization to better capture structural information.
Authors
Alessio Colombo, Melika Ayoughi
Abstract
Residual Vector Quantization turns continuous representations into discrete, multi-level token sequences. Yet most methods operate in Euclidean space, despite the coarse-to-fine structure of the resulting codes and the latent hierarchies present in many data domains. Hyperbolic geometry offers a natural alternative for hierarchical representations, but naive hyperbolic extensions introduce geometric inconsistencies: non-associative hyperbolic addition prevents consistent residual aggregation, while standard straight-through gradient estimation ignores the geometry of the latent space. We propose a geometry-aware hyperbolic residual quantization that addresses these issues in both the forward and backward passes. In the forward pass, Hyperbolic Residual Aggregation restores the telescoping behavior of residual quantization on the Poincare ball. In the backward pass, a discounted Hyperbolic Straight-Through Estimator routes the reconstruction gradient through the quantizer as a single geometric block, avoiding unstable recursive gradient transport across residual stages. Evaluations on hierarchical prediction, recommendation, image tokenization, and neural audio coding tasks show that our method improves the stability and structural organization of hyperbolic residual codes over naive hyperbolic baselines. At the same time, we observe a clear structure-compression trade-off: Euclidean residual quantization remains preferable for pure compression, while geometry-aware hyperbolic quantization is most useful for hierarchically organized discrete latent spaces.