Phasor Attention: Mean Root Square Normalization for Phase Manifold Preservation
2026-07-20 • Machine Learning
Machine LearningArtificial Intelligence
AI summaryⓘ
The authors identify problems with a common technique called Root Mean Square Normalization used in deep learning, which can cause instability during training. They propose a new method called Mean Root Square Normalization (MRSNorm) that pairs data channels into 2D groups and changes the way normalization is calculated to keep values more stable. This new approach reduces the number of parameters by half and improves training stability, especially in difficult conditions where traditional methods fail. Their tests show MRSNorm prevents numerical issues and keeps training stable without extra complexity.
Root Mean Square Normalizationnormalizationgradient clippingphasoraffine transformationgradient homogeneitydeep learning optimizationResNetCIFAR-100numerical stability
Authors
Sungwoo Goo, Hwi-yeol Yun, Sangkeun Jung
Abstract
While Root Mean Square Normalization has become the de facto standard for accelerating modern sequence models, its reliance on the quadratic accumulation of independent scalars ($\sum x^2$) inherently triggers outlier-induced numerical instability, gradient starvation, and anisotropic phase distortion. We introduce Mean Root Square Normalization (MRSNorm). By structurally pairing channels into 2D phasors, MRSNorm mathematically inverts the traditional scaling paradigm: it computes the localized $L_2$ magnitudes (Root Square) before aggregating them via a global $L_1$ average (Mean). This operational inversion strictly constrains activations to a phasor manifold, preserving conformal invariance. By sharing a single affine weight across phasor components, MRSNorm halves the total number of learnable parameters, proving that unconstrained spatial scaling in standard norms is a harmful redundancy. We analytically demonstrate that this geometric constraint yields a built-in, trigonometric gradient clipper governed by the Pythagorean identity, unconditionally equalizing the local gradient norm to ensure Gradient Homogeneity. Empirical evaluations on a ResNet with CIFAR-100 show that despite halved parameters, MRSNorm provides critical structural stability under rigorous stress tests. Under extreme hyperparameter settings where standard normalizations suffer from gradient divergence, MRSNorm successfully prevents numerical explosion and secures stable optimization trajectories. Our findings propose a fundamental paradigm shift toward phasor-based deep representation learning. The implementation of MRSNorm is available at Appendix C.