Orbit compresses large language model embeddings with learned geometry
OrBIT: Structure-Guided Embedding Compression
Machine Learning
Summary
Large language models use tables of numbers called embeddings that take up a lot of space. This paper introduces OrBIT, a method that learns special geometric patterns in these tables to compress them more efficiently. Unlike previous methods that fix a certain compression style, OrBIT discovers and uses reusable local structures to store the data compactly. The authors show that this approach can reduce the size of embedding tables by over 20 times while keeping their quality nearly as good as less compressed versions.
What this means in practice
- •For large-scale ai developers: Reduce memory and storage costs for large language model embeddings using learned geometric compression.
- •For data center operators: Decrease bandwidth and storage demands in serving language models by applying compact embedding representations.
Authors
Yunied Puig, Amit Kumar Jaiswal
Abstract
Embedding tables are among the largest components of modern language models. Most compression methods fix a coding geometry such as coordinate blocks, low-rank subspaces, or unrestricted codebooks, and optimize within it. We instead ask whether the coding geometry can itself be discovered. We introduce \emph{OrBIT}, a structure-guided embedding compression framework that learns reusable local geometry from orbit dynamics and uses it to constrain a small set of shared codewords. The global reconstruction residual then decides where the fixed coding budget is spent, while redundant overlapping charts let local errors compensate one another after gluing. Our theory shows how tight-chart geometry controls distortion, how the global residual directs sequential allocation, and how data-geometry-guided refinement improves the codec. The resulting orbit machinery is compiled away, leaving a compact decoder in which the learned structure governs what is stored, where capacity is allocated, and how local information is assembled globally. Across four LLM embedding tables, OrBIT achieves $37.9\times$ compression on GPT-2 and over $23\times$ on each 7B table relative to 16-bit storage, while delivering competitive rate-distortion performance against established quantization and low-rank baselines.