LaMoC: Loss-Aware Modular Compression for LLMs
2026-08-31 • Artificial Intelligence
Artificial IntelligenceComputation and LanguagePerformance
AI summaryⓘ
The authors studied how to compress large language models by breaking them into parts (modules) without losing accuracy. They found that past methods mainly used data about the models' activations but ignored how sensitive each part is to errors in terms of the model's overall learning loss. Their new method, LaMoC, combines activation information with a loss-aware measure called the Empirical Fisher, better matching compression with the model’s performance impact. Experiments on various models showed LaMoC reduces prediction errors and slightly improves task accuracy compared to previous methods.
modular compressionlarge language modelsactivation statisticsEmpirical Fisherloss sensitivityperplexitygradient-error alignmentmodule reconstruction errortwo-tiered optimization
Authors
Mohanad Odema, Jacob Song
Abstract
Modular compression has enabled considerable parameter reduction in LLMs while preserving strong language understanding and downstream task accuracy. However, existing joint modular compression methods primarily rely on activation statistics, leaving loss-sensitivity information and its module-level characterization underexplored. We investigate addressing this gap with LaMoC, a loss-aware modular compression methodology that blends activation and Empirical Fisher statistics through gradient-error alignment. LaMoC improves joint compression by selecting compression statistics that better align local module reconstruction error with the downstream loss. Our contributions are three-fold: (1) We characterize the Empirical Fisher as a module-level loss-aware proxy that can be blended with the activation statistics required for compression. (2) We reformulate joint modular compression as a two-tiered optimization problem that minimizes module reconstruction error while tuning the activation and gradient information blending rate. (3) We implement an empirically driven methodology with statistical validation to solve the resulting compression problem. We evaluate LaMoC across four model families spanning eight models. On the 4-8B models, LaMoC achieves an average 2.5% reduction in perplexity and a 1% relative improvement in task accuracy over state-of-the-art modular compression methods.