LinQuant improves low-precision inference for linear recurrence models
LinQuant: Functional Invariances for Low-Precision Linear Recurrences
Machine Learning
Summary
Large language models often use special math operations called linear recurrences to handle sequences of information efficiently. But running these operations quickly on low-precision (less detailed) hardware is tricky and can lead to mistakes. The authors studied properties of these operations that don’t change their results, called invariances, and used them to adjust the math so it works better with low-precision computing. They created LinQuant, a method that applies these tweaks to improve accuracy without slowing things down, tested successfully on several big language models. This helps make language AI faster and more efficient on hardware with limited precision.
What this means in practice
- •For machine learning engineers: Enhance the efficiency and accuracy of linear recurrence-based language models on low-precision hardware during inference.
- •For hardware accelerator designers: Design accelerators that better support quantized matrix operations using invariance transformations for faster language model inference.
Authors
Neehal Tumma, Philipp Nazari, Rohan Cherukuri, Daniela Rus, T. Konstantin Rusch
Abstract
Linear recurrences are emerging as practical alternatives to softmax attention in large language models, appearing in both standalone and hybrid architectures. However, despite their rapid emergence, low-precision inference remains much less developed for these models than for softmax attention. During prefill, chunkwise-parallel algorithms cast much of the recurrence as dense matrix multiplications, suggesting a natural path to low-precision execution. Simply quantizing these operations in their native form, however, can be fragile. In this work, we identify functional invariances that preserve the full-precision computation while making the operands more amenable to quantization. Building on this observation, we introduce LinQuant, a framework for low-precision recurrent prefill based on seven such invariances, including rotations, diagonal rescalings, and alternative placements of recurrent decay and write factors. We derive when these transformations remain valid across common recurrence families spanning linear, gated, and delta-rule updates, and show that they provide complementary degrees of freedom for improving low-precision accuracy. Finally, we translate this framework to production by augmenting SSDi8, an existing low-precision prefill pipeline for Mamba-2, with the applicable invariances. LinQuant improves perplexity at matched latency on Mamba-2 1.3B, 2.7B, and 8B and Nemotron-H 8B, demonstrating the practical utility of invariance-based quantization for recurrent models.