Spectral pruning reduces Transformer attention size without hurting accuracy
Prescriptive SVD-Inspired Attention via Spectral Energy Retention
Machine LearningComputer Vision and Pattern Recognition
Summary
Transformer models use a method called self-attention to understand information, but it's hard to tell which parts of this method really matter. The authors study a version called SVD-Inspired Attention (SVDA) that breaks down attention into measurable parts. They show that by keeping only the most important parts of this breakdown, the model can become smaller and simpler without losing accuracy. This makes it easier to understand and improve how attention works inside these models.
What this means in practice
- •For machine learning engineers: Reduce Transformer model sizes by pruning attention score directions while maintaining accuracy for efficient deployment.
- •For mobile device developers: Deploy slimmer Transformer models with fewer computations and parameters for on-device AI that saves power and memory.
Authors
Vasileios Arampatzakis, Vasileios Sevetlidis, George Pavlidis
Abstract
Self-attention is central to modern Transformer architectures, but its dense dot-product formulation makes it difficult to identify which internal directions are structurally important and which can be modified without disrupting the model. SVD-Inspired Attention (SVDA) addresses part of this problem by introducing a learned diagonal spectrum into the query-key score interaction, making latent attention directions explicitly inspectable through indicators such as spectral entropy, effective rank, sparsity, alignment, selectivity, and perturbation response. This paper examines the transition from diagnostic interpretation to operational intervention. A diagnosis--intervention--verification framework is proposed, and one intervention is evaluated: spectral energy retention in the attention-score pathway. Across FashionMNIST, CIFAR-10, CIFAR-100, and Food-101, the $ρ=0.90$ prescription removes 24.5--53.7\% of score directions, reduces parameters by 2.6--4.3\%, and reduces estimated MACs by 2.8--5.4\%. The paired mean accuracy change of the dimension-reduced model ranges from $-0.03$ to $+0.05$ percentage points over three seeds. These results support SVDA as an intrinsically interpretable attention mechanism whose learned spectrum exposes an operational coordinate system for deterministic and verifiable modification of attention-score formation.