After the Euclidean Highway: Hyperbolic Expert AI as the Next Innovation
2026-07-20 • Machine Learning
Machine LearningArtificial IntelligenceComputation and Language
AI summaryⓘ
The authors studied how to best use hyperbolic geometry in neural networks for expert systems that have a tree-like structure. They found that applying hyperbolic geometry only at the loss layer, rather than throughout the network or in adapters, leads to stable and successful training. Their method, called HySAT, avoids many training failures seen in other approaches. They tested this across multiple large models and provided proofs explaining why this focused placement works well. Some of their models are publicly available and even live in consumer applications.
hyperbolic geometrytransformersloss layerexpert systemsmanifold invariantsadapter trainingtree structuretraining stabilityLlama 3.1SLM (specialized language models)
Authors
Kwan Soo Shin, In Seok Kang, Munho Lee
Abstract
Expert domains are trees; the Euclidean transformer is not, diluting parent-child structure exponentially at depth. The hyperbolic turn left one question unasked: not how much of a network to curve, but where curvature may touch the gradient. Placement is a law, not a knob: the same geometry on a trainable adapter collapses training (seventeen training collapses, ~220 GPU-hours), yet at the loss layer alone it trains without one -- this is HySAT (Hyperbolic Structure-Aware Training), hyperbolic losses at the loss layer only. Across six expert SLMs we constructed and deployed (Llama 3.1 and EXAONE 3.5; four adapter strategies; 18.0M-sample corpus; zero NaN over ~317K optimizer steps), a matched four-arm ablation isolates the preserved manifold invariant, and three propositions and a lemma prove why loss-only placement is stable where adapter-on-manifold is not. Four models are operationally deployed (one live, consumer-facing), two open-weight, with per-step traces and a seventeen-incident failure ledger on Zenodo (CC-BY-4.0).