Layer compensation sometimes reduces neural network quantization errors
When Does Compensating Inherited Quantization Error Help? Mean-Square Guarantees for Layer-Wise Quantization and Their Limits
Machine LearningArtificial IntelligenceInformation Theory
Summary
Quantizing a neural network means simplifying its numbers to save space but can cause errors. The paper studies when allowing each layer to fix errors inherited from earlier layers actually reduces the final error in the network’s output. The authors show that compensation helps under certain ideal mathematical conditions, such as in linear networks with infinite number codes, but often fails with real networks that have nonlinear parts or limited codebooks. They also test this on large models and find that partial compensation usually helps, yet no general rule picks the best amount of compensation all the time.
What this means in practice
- •For machine learning engineers: Improve post-training quantization by tuning layer-wise compensation to reduce output errors in large neural networks.
- •For chip designers: Design quantized neural network accelerators that incorporate compensatory feedback to minimize accumulated rounding noise.
Authors
Tianhao Guo
Abstract
Layer-wise post-training quantization methods let each layer compensate for the quantization error inherited from earlier layers. We ask under what assumptions such compensation guarantees a network output error no larger than without it. For linear networks of any depth, with the required calibration Gram matrices invertible, if the rounding error of each layer has zero conditional mean and a deterministic second-moment map common to both schemes, as under subtractive dither on an unbounded grid, then compensation with strengths in $[0,1]$ fixed in advance gives a mean-square output error no larger than no compensation, each scheme being followed along its own trajectory. The exact gain separates error removed directly from reduced excitation of later rounding noise. Outside this setting the comparison can fail. With rectified linear activations it fails for two independent reasons: noise changes activation states, and, even when no state changes, the correction moves error from samples on which a unit is inactive to samples on which it is active. With stable activations, commutators between the calibration projection and the activation patterns bound the excess. With a bounded codebook, even unique exact layer-wise minimizers with no overload can give a larger output error. For fixed rounding errors it already fails for two scalar linear layers, refuting a published universal guarantee. An exact residual identity gives the necessary and sufficient condition on a common layer input. All these comparisons hold the calibration input fixed. On two 8B models at 4 bits, fixed half-strength compensation beat uncompensated rounding in every model, text and draw tested, yet no rule for choosing the strength beat a layer-wise rule consistently, and lowering the full-network loss on the data used to choose did not reliably lower it on new text.