Fisher-weighted svd compression improves vision transformer efficiency
Mind the Approximation: Fisher-Weighted SVD Compression for ViTs
Computer Vision and Pattern RecognitionArtificial IntelligenceMachine Learning
Summary
Large machine learning models, like Vision Transformers used in image tasks, can be too big and slow for easy use. The authors show that a common way to shrink these models using a math trick called Fisher-weighted SVD doesn't always predict how well the smaller model works. They create a new method, FACTS, that better compresses these models by focusing on how parts of the model work locally. They also develop a way to smartly decide how much to shrink each part, leading to smaller models that keep accuracy without extra training.
What this means in practice
- •For machine learning engineers: Shrink Vision Transformer models for deployment by improving compression accuracy without retraining.
- •For computer vision developers: Increase efficiency of image recognition systems by applying FACTS to reduce model size while maintaining accuracy.
Authors
Moritz Thoma, Maximilian Groezinger, Maximilian Forstenhäusler, Emad Aghajanzadeh, Ryan Pegoud, Manoj Rohit Vemparala, Pierpaolo Mori, Alexander Frickenstein, Daniel Mueller-Gritschneder, Ulf Schlichtmann
Abstract
Model compression is key to mitigate deployment challenges of ever growing machine learning models. In this area of research, singular value decomposition (SVD)-based compression offers a compelling trade-off between computational efficiency and model accuracy. Fisher-weighted SVD in particular provides principled, loss-aware compression. However, we find that improving the fidelity of Fisher approximation used in the compression is poorly predictive of post-compression accuracy for Vision Transformers (ViTs). Motivated by this observation, we propose FACTS, a structured Fisher Approximation tailored to Compressing ViTs with Fisher-weighted SVD, which enforces token-local aggregation while preserving within-token activation-gradient dependence. Additionally, we introduce a fast Constrained Rank Search (CoRS), that optimizes layer-wise rank allocation while adhering to a fixed floating point operation (FLOP) constraint. Extensive experiments across ViTs and hybrid architectures demonstrate that FACTS consistently improves accuracy-efficiency trade-offs without requiring finetuning. Notably, it outperforms the strongest SVD baseline by up to +5.8 percentage points (p.p.) Top-1 on Swin-B, with further gains driven by our search method. Code is available at https://github.com/MoritzTho/FACTS.