Small language model compression tested for accuracy and speed trade offs
Compression Technique Interactions in Small Language Models: A Factorial Ablation Study
Artificial Intelligence
Summary
Making smaller language models run on limited hardware often means shrinking them, but it’s not clear how different shrinking methods work together. The authors studied many combinations of methods like quantization (reducing the bit size), pruning (removing parts), and recovery tweaks. They found that some techniques, like 8-bit quantization, work well without hurting performance, while others can cause problems. They also showed that a fine-tuning method called LoRA helps fix quality drop from heavy pruning. Their tests cover different models and hardware, including CPUs.
What this means in practice
- •For mobile app developers: Run compressed small language models efficiently on devices with limited memory using 8-bit quantization without accuracy loss.
- •For edge device engineers: Optimize CPU-based inference speed by choosing between 4-bit or 8-bit model quantization based on speed versus small accuracy differences.
Authors
Bhushit Nema, Siddharth Krishnamurthy, Anjali Kadiyala
Abstract
Deploying small language models on resource-constrained hardware requires compression, yet practitioners lack systematic guidance on how quantization, pruning, and post-compression recovery interact when combined. We present a factorial ablation study evaluating 72 compression configurations across four SLM architectures (2.7B-7B parameters), extended with a 48-configuration ordering analysis, benchmark accuracy validation on 30 compressed models, and robustness checks including multi-seed validation and full-corpus perplexity evaluation. INT8 quantization is effectively lossless (<0.5% accuracy degradation), while NF4 quantization combined with L1 magnitude pruning causes model collapse in all 16 tested combinations. LoRA-based fine-tuning reduces degraded perplexity by up to 97% after aggressive pruning; an ablation against standard fine-tuning without a teacher model shows the recovery is driven by LoRA adaptation itself rather than the distillation signal. Compression ordering shows no significant effect among viable configurations (ANOVA p = 0.96) but determines viability for NF4+pruning combinations. INT8 quantization alone provides 40-45% memory reduction with no quality loss. On-device CPU deployment via llama.cpp confirms these findings transfer off the GPU: 4-bit quantization runs 46-54% faster than 8-bit for a perplexity difference under 0.5 points, while un-structured pruning yields no CPU latency or memory benefit, consistent with its dense storage format.