Adaptive Model Compression (AMC): Saliency-Driven Resource Allocation for Ultra-Low-Power Transformer Inference
2026-07-11 • Information Retrieval
Information RetrievalArtificial IntelligenceHardware ArchitectureMachine Learning
AI summaryⓘ
The authors address the problem of running large transformer models on devices with limited power and memory by creating a system called Adaptive Model Compression (AMC). Their method identifies important parts of the data to process fully, while simplifying less important parts to save energy and speed up processing. Tests show AMC uses much less energy and runs faster with only a small drop in accuracy. This helps extend battery life on mobile devices by using full computing power only when really needed.
transformer modelsedge devicesadaptive compressionsaliency-driven processingbit-width reductionhardware resource allocationsystem energythroughputmodel accuracymulti-tier architecture
Authors
Jiayin Hu, Kai Yuan, Vanessa Hu, Xuetao Yin, Jianhua Li, Sean Suchter
Abstract
Deploying large-scale transformer models on resource-constrained edge devices remains a challenge due to the high energy and memory overhead inherent in static inference, which processes simple and complex tokens with uniform intensity. To address this, we propose Adaptive Model Compression (AMC), a saliency-driven framework that dynamically allocates hardware resources based on token importance. By implementing a multi-tier architecture, our system identifies critical high-saliency information for full-precision processing while aggressively reducing the rank and bit-width of less significant data. Experimental results demonstrate that AMC achieves a 59.2% reduction in system energy and a 2.24x increase in throughput on 45nm CMOS hardware. This approach effectively extends the battery life of mobile devices by utilizing high-definition compute only where necessary, maintaining robust performance with a marginal 3.6% accuracy trade-off.