Global attention rankings stay stable despite weight quantization shifts
Global Ranks Survive, Selected Heads Shift: BOS-Sink Topology under 4-bit Weight-Only Quantization
Machine LearningComputation and Language
Summary
When large language models are compressed to use less memory, some parts that focus on important words may change. The authors tested how well strategies to find key parts before compression still work after making the model much smaller with 4-bit weights. They found that while the overall importance rankings of attention heads remain similar, the exact selection of top heads can change notably, especially in certain layers and when the language style shifts. This means systems should check their focus settings again after compression instead of assuming they stay the same.
What this means in practice
- •For machine learning engineers: Ensure recalibration of key attention mechanisms after 4-bit quantization to maintain model accuracy in deployed language models.
- •For edge ai developers: Use validated global ranking maps to deploy compressed language models on devices with limited resources while monitoring layer-specific shifts.
Authors
Kuanlin Chen, Chen-Wei Kuo, Cheng-En Ou
Abstract
Sink-aware deployment may identify important first-token attention heads before a model is quantized, then reuse that map at the edge. We test when this shortcut is safe for 4-bit NF4 weight-only post-training quantization (PTQ). Our Sink Topology Consistency (STC) metrics separate global rank preservation, top-$k$ set overlap, and layerwise sink-mass shift, and distinguish per-input sensitivity from calibration-map transfer. Across Qwen2.5-0.5B, Qwen2.5-1.5B, and Llama-3.2-1B, global bf16-to-4-bit ranks remain high at 4,096 tokens ($ρ_s \geq 0.980$), yet top-$k$ Jaccard overlap is only 0.619-0.793, corresponding to 76.5-88.5% membership retention. The global statistic also masks local failures: terminal Qwen layers shift by 6.2-7.9x their model means, whereas Llama-3.2-1B shows low, nearly uniform drift. Under a C4-to-LongBench shift, cross-domain overlap degrades more than the within-domain precision comparison for both Qwen models, but not for Llama-3.2-1B. Matched-domain 4-bit recalibration reaches 90% of a split-half stability plateau at the smallest tested $n=8$ for both Qwen models and $n=32$ for Llama-3.2-1B, though not as a sharp threshold; for the two Qwen models, updating only selected layers does not reach the full-map stability criterion. On Jetson Orin NX, the 16-sample workload takes seconds for the two models with valid on-device sink measurements. The practical message is precise: global rankings often transfer, but discrete head sets, layer-local policies, and cross-domain calibration should be revalidated after quantization.