Low-rank models show varied recovery patterns after compression
The Functional Structure of Post-Compression Recovery in Low-Rank LLMs
Artificial IntelligenceInformation Theory
Summary
When large language models (LLMs) are compressed to save space, they can behave quite differently during the fix-up process afterward. The authors found that these differences are linked to hidden functional details they call recovery pressure. They studied this pressure across different parts of the model and tracked how it changes during the recovery, showing that it influences how well the model improves. This understanding gives a new way to measure recovery progress beyond just looking at overall error.
What this means in practice
- •For machine learning engineers: Improve fine-tuning strategies by monitoring module-wise recovery pressure to better restore compressed language models.
- •For ai model maintainers: Track evolving functional changes during recovery to predict and optimize post-compression model performance upgrades.
Authors
Zishan Shao, Liang Tian, Georgiy Zemlevskiy, Kangning Cui, Lixun Zhang, Yixiao Wang, Ting Jiang, Jinhee Kim, Yixuan Chen, Rui-Feng Wang, Fan Yang, Hai Li, Yiran Chen
Abstract
Different low-rank compression methods can produce compressed LLMs that respond differently to the same post-compression recovery procedure, and relative advantages observed between methods at the compression endpoint may shrink, grow, or even reverse after recovery. We ask whether this recovery heterogeneity reflects functional structure beyond scalar loss evolution, and how that structure evolves throughout recovery. Our results establish that this heterogeneity reflects a reproducible compression-induced functional structure, which we formalize as recovery pressure. To characterize this structure consistently throughout recovery, we develop a standardized functional characterization within each backbone that is applicable across heterogeneous low-rank methods. The primary backward characterization reveals reproducible module-wise structure across independent probes, while a complementary forward-only characterization recovers related structure without loss or backpropagation. We further find that recovery pressure measured at the endpoint is associated with subsequent recovery response; during recovery, its module-wise structure is reorganized non-uniformly, and localized updates induce distributed responses beyond directly updated modules. Further evidence indicates that tracking this evolving structure provides a complementary functional view of recovery progress alongside scalar loss.