Improved method cuts language model compression cost and boosts stability
Structured Transforms for Low-Overhead Quantization of Language Models
Computation and Language
Summary
Large language models use many numbers that take up a lot of space and slow down computations. The authors revisit a way to shrink these numbers while keeping model quality by breaking each number into two parts with simple properties. They replace a costly step with a faster math trick using the Discrete Cosine Transform, making the process much quicker. This approach also avoids instability problems found in other methods, keeping the model reliable even in tough situations. The improvements mean smaller, faster, and more stable versions of large language models are possible without complicated retraining.
What this means in practice
- •For machine learning engineers: Compress large pretrained language models quickly and reliably for faster inference on hardware supporting low-bit operations.
- •For ai hardware developers: Design native 2-bit compatible weight storage schemes leveraging the structured factorization supporting efficient decoding during inference.
Authors
Daria Cherniuk, Alexander Rudikov, Boris Kashin, Ivan Oseledets
Abstract
We revisit Kashin-decomposition-based weight quantization for large language models and propose an improved algorithm with stronger convergence properties and structured, efficient orthogonal transforms. The method retains the core factorization of each weight into two components -- one with bounded infinity norm and the other with bounded infinity norm after an orthogonal transformation -- but replaces the dense random orthogonal matrix with a sign-randomized Discrete Cosine Transform (DCT), reducing the per-iteration cost from $\mathcal{O}(N^2)$ to $\mathcal{O}(N \log N)$. The proposed greedy algorithm with alternating updates guarantees the four-peak distribution required for stable 2-bit clustering of each factor and admits closed-form initialization of cluster centers, removing the multi-restart k-means bottleneck of prior work. Composed with OPTQ-style sequential error compensation and QuIP-style incoherence preprocessing, the resulting JAX pipeline is competitive with OPTQ, QuIP, QuIP-RG and a fine-tuning- and vector-quantization-free variant of QuIP# at 4-bit per channel on OPT, Llama-2 and Pythia, with favorable wall-clock scaling. The bounded-$\ell_\infty$ factorization is also notably robust: on stress configurations where QuIP variants diverge to four-digit perplexity (Pythia-6.9B) or abort with NaNs in LDL back-substitution (Mistral-7B), Kashin-DCT remains numerically stable and stays close to FP16 baseline. At inference time, each weight decomposes into two 2-bit factor codes per channel that are structurally suited to native-2-bit hardware.