Massive Spikes in LLMs are Bias Vectors: Mechanistic Uncovering and Spike-Free Quantization
2026-06-01 • Machine Learning
Machine Learning
AI summaryⓘ
The authors studied weird, sudden spikes in Large Language Models that make it hard to shrink model size without losing accuracy. They found these spikes come from fixed hidden vector patterns in specific tokens, not just random scalar values. By understanding how the model's internal parts handle these vectors and how the model protects them, the authors created a new way to limit spikes during quantization, called INSERTQUANT. This method helps keep the models accurate even when using fewer bits, and it works not only for text models but also for vision models. Their technique matches top existing methods while being simpler and more general.
Large Language Modelsquantizationactivation spikesvector biasesattention mechanismRotary Positional Embedding (RoPE)post-training quantizationINSERTQUANTlow-bit quantizationVision Transformers (ViTs)
Authors
Yung-Chin Chen, Chung Peng Lee, Ze-Wei Liou, Naveen Verma
Abstract
Massive activation spikes in Large Language Models (LLMs) severely degrade quantization by stretching dynamic ranges. While prior hypotheses characterize these as high-level scalar biases, we argue that they are merely the scalar intermediates of rigid, structural vector biases in the spike-carrying tokens. We show that these tokens converge to constant vectors after normalization that drive the attention sink and value-state drain mechanisms. We geometrically substantiate this by analyzing the coordination of projection weights: $W_K$ contrastively amplifies the vector, $W_Q$ aligns semantic tokens toward it, and $W_V$ projects it into the spectral null-space. Furthermore, we reveal that the model actively preserves these structural biases against Rotary Positional Embedding (RoPE) perturbations by localizing them in "zones of rotational stability" utilizing low-frequency bands and coherent channel pairs. Leveraging this, we propose INSERTQUANT, a post-training quantization (PTQ) framework that clamps spikes and restores their function via pre-computed template vectors. This renders activations strictly spike-free, enabling robust low-bit quantization with high fidelity. INSERTQUANT achieves parity with state-of-the-art per-tensor quantization methods on LLMs and uniquely generalizes beyond text to other modalities such as ViTs.