Spectral weight decay improves neural network compression and accuracy

Spectral Weight Decay: Inducing Low-Rank Structure in Neural Network Weights

Machine Learning

Summary

Neural networks learn by adjusting many numbers called weights, but these weights often have hidden patterns that regular methods ignore. The authors introduce a new way to gently shrink these weights by focusing on their underlying structure, making the network simpler without losing accuracy. This method helps compress the model, speeding up its use, and improves how well networks learn from noisy data. They tested this on popular language and vision models and saw clear benefits.

What this means in practice

  • For machine learning engineers: Compress large language models to reduce size and speed up inference without sacrificing accuracy by using spectral weight decay.
  • For ai model trainers: Improve robustness and final accuracy of neural networks trained on noisy data by applying spectral weight decay during training.

Authors

Dmitrii Andriianov, Andrey Veprikov, Aleksandr Beznosikov

Abstract

Standard weight decay treats each weight matrix as a vector and ignores its spectral structure. We introduce spectral weight decay, a post-step decoupled nuclear-norm update that applies additive rather than multiplicative spectral shrinkage. We connect the update to approximate proximal descent and show that its sensitivity to update order can exceed that of conventional $\ell_2$ weight decay near rank deficiency. Across LLaMA models with $124$M to $500$M parameters, spectral weight decay lowers effective rank and improves SVD-LLM compression at matched validation loss. At $500$M and a $4\%$ distortion budget, it reaches $1.89\times$ compression and $1.18\times$ GPU inference speedup, compared with $1.14\times$ and $1.01\times$ after standard weight decay. Under fixed-horizon training with $60\%$ label noise, it also improves final mean clean-test accuracy over matched $\ell_2$ regularization by up to $17.8$ points on MNIST and $4.6$ points across four BERT-base tasks. Code is available at https://github.com/brain-lab-research/SpectralWD.