Beyond Activation Alignment:The Alignment-Diversity Tradeoff in Task-Aware LLM Quantization
2026-07-01 • Machine Learning
Machine Learning
AI summaryⓘ
The authors studied how to better compress large language models using mixed-precision quantization, which reduces the number of bits used to store model parameters. They found that measuring layer importance by how it affects perplexity doesn't predict how well the model does on complex reasoning tasks. They also discovered that mixing different types of calibration data improves the robustness of quantization. Based on this, they created TASA, a method that smartly chooses calibration data and allocates bits to different parts of the model. Their tests showed TASA can produce smaller models that perform as well or better than some bigger, less carefully quantized ones.
Mixed-precision quantizationPerplexityCalibration dataSensitivity analysisBit allocationLarge language modelsReasoning tasksModel compressionLLaMAGSM8K
Authors
Fei Wang, Chao Xue, Taoran Liu, Li Shen, Ye Liu, ChangXing Ding
Abstract
Mixed-precision quantization (MPQ) has become a key technique for deploying large language models under stringent memory and compute constraints. We first identify a phenomenon that we term the Perplexity Illusion: layers ranked as important by perplexity-based sensitivity show little rank correlation with those that are most influential for complex reasoning performance, with Kendall $τ\approx 0$ in our analysis. We further reveal an Alignment-Diversity Tradeoff: using only target-task calibration data can degrade post-quantization performance, whereas incorporating general-domain data stabilizes sensitivity estimation and improves robustness across tasks. Based on these observations, we propose TASA (Task-Aware Sensitivity Analysis), a two-level framework that jointly optimizes calibration-data composition and mixed-precision bit allocation. Specifically, TASA searches for a calibration-data mixture using a training-free gradient-trace alignment criterion, and then aggregates perplexity and reasoning-oriented sensitivity signals to guide both inter-layer and intra-layer bit allocation. Experiments on LLaMA-3-8B and Qwen2.5-7B reveal a precision inversion: appropriately allocated 3.5-bit models can match or surpass less task-aware 4-bit baselines. At an average precision of 3.5 bits, TASA matches or outperforms several competitive 4-bit uniform baselines in aggregate accuracy, and improves over the strongest W3 baseline on GSM8K by more than 20 absolute points on LLaMA-3-8B. These results show that calibration-data composition substantially affects task-sensitive quantization, a factor underexplored in prior work.