Activation quantisation breaks retrieval computations more than induction
Perplexity Cost Understates What Activation Quantisation Breaks
Machine Learning
Summary
When compressing parts of AI models called activations into fewer bits, a common measure called perplexity shows a general sense of quality loss. However, the authors found that this average score hides how specific types of model operations, especially retrieval, get damaged much more than others like induction. They discovered that the pattern of errors depends on the direction and sign of quantisation errors, not just how big the errors are. By changing the way these errors align with internal model calculations, they could restore much of the induction ability but retrieval recovered less well.
What this means in practice
- •For machine learning engineers: Improve activation quantisation methods by considering error directionality to better preserve induction and retrieval computations in language models.
- •For ai hardware developers: Design quantisation hardware that supports rotated coordinate systems to mitigate damage to specific computations in compressed activation representations.
Authors
Anish Sathyanarayanan
Abstract
Activation quantisation is usually evaluated with an aggregate metric, perplexity, averaged over every token a model predicts. We ask whether that average identifies which computations a quantiser damages. Perplexity turns out to be a reliable aggregate signal: across 12 models from four families and 780 within-model comparisons, the arm perplexity prefers also retains more induction and more retrieval in all but 2.1 and 4.0 percent of cases respectively. But where perplexity has risen by only a factor of 1.2 to 1.5, induction still keeps 0.959 of its intact accuracy while retrieval has already fallen to 0.554, a gap the aggregate number does not surface. This gap has structure, not just size: a matched Gaussian-noise control of the same per-channel magnitude leaves it largely intact, and randomising only the sign of the quantisation error, every magnitude held fixed, is nearly as harmless, so magnitude alone does not explain the damage. Quantising in a rotated basis, which changes coordinate alignment without changing error magnitude, restores induction from 0.001 to 0.980 at three average bits per token in a single-block intervention, though retrieval recovers less completely at the same setting (0.694); end-to-end at four average bits, induction reaches 0.968 and retrieval 0.534. The pattern holds on two further models up to 32B parameters and, in the deployed configurations we tested, under AWQ once activations are pushed to 4 bits. A perplexity target bounds the average cost of a transformation applied to the activation; it does not, by itself, show which computations survived.