Vision transformer models get better compression with smarter math
Mind the Approximation: Fisher-Weighted SVD Compression for ViTs
Computer Vision and Pattern RecognitionArtificial IntelligenceMachine Learning
Summary
Large vision transformer models are hard to run efficiently, so researchers try to compress them while keeping their accuracy. The authors found that a common way to improve the compression math does not always lead to better model results. To fix this, they designed a new method called FACTS that better matches the model’s internal behavior and keeps important connections during compression. They also created a way to pick the best settings for each layer to use less computation. Their approach improves accuracy without extra training and beats previous methods on several popular models.
Model compressionSingular value decomposition (SVD)Vision transformers (ViTs)Fisher informationToken-local aggregationActivation-gradient dependenceFloating point operations (FLOPs)Rank allocationTop-1 accuracyFinetuning
Authors
Moritz Thoma, Maximilian Groezinger, Maximilian Forstenhäusler, Emad Aghajanzadeh, Ryan Pegoud, Manoj Rohit Vemparala, Pierpaolo Mori, Alexander Frickenstein, Daniel Mueller-Gritschneder, Ulf Schlichtmann
Abstract
Model compression is key to mitigate deployment challenges of ever growing machine learning models. In this area of research, singular value decomposition (SVD)-based compression offers a compelling trade-off between computational efficiency and model accuracy. Fisher-weighted SVD in particular provides principled, loss-aware compression. However, we find that improving the fidelity of Fisher approximation used in the compression is poorly predictive of post-compression accuracy for Vision Transformers (ViTs). Motivated by this observation, we propose FACTS, a structured Fisher Approximation tailored to Compressing ViTs with Fisher-weighted SVD, which enforces token-local aggregation while preserving within-token activation-gradient dependence. Additionally, we introduce a fast Constrained Rank Search (CoRS), that optimizes layer-wise rank allocation while adhering to a fixed floating point operation (FLOP) constraint. Extensive experiments across ViTs and hybrid architectures demonstrate that FACTS consistently improves accuracy-efficiency trade-offs without requiring finetuning. Notably, it outperforms the strongest SVD baseline by up to +5.8 percentage points (p.p.) Top-1 on Swin-B, with further gains driven by our search method. Code is available at https://github.com/MoritzTho/FACTS.