Layer normalization alters neural network width and approximation needs
Activation-Adjacent Layer Normalization: Effects on Approximation Power
Machine Learning
Summary
Layer normalization and a related method called RMSNorm change how neural networks behave by affecting their internal calculations during use. The authors studied how these normalizations influence a network's ability to mimic other simpler networks of fixed size. They found specific minimal sizes (widths) needed for networks using these normalization methods to approximate others, depending on details like activation functions. Their results provide formulas and proofs about these requirements and confirmed the findings through experiments.
What this means in practice
- •For deep learning engineers: Design neural network architectures using normalized layers with minimal width ensuring desired approximation accuracy.
- •For machine learning tool developers: Improve frameworks by incorporating approximation bounds for networks with layer normalization and RMSNorm during model validation.
A theory result. No direct application yet.
Authors
Yunhao Ni, Yuxin Guo, Jie Luo, Wenjun Wu, Lei Huang
Abstract
Layer normalization (LN) and root mean square normalization (RMSNorm) depend on their inputs and remain nonlinear at inference. Inserting either next to an activation may therefore change the function a network represents. We study when shallow activation-normalized (AN) and preactivation-normalized (PN) networks can approximate a prescribed width-$N$ one-hidden-layer MLP. Viewing RMSNorm as input-dependent scaling, we use width augmentation to locally linearize normalization on compact feature sets. For AN networks, an unbounded activation range allows width $N+1$, while every continuous nonconstant activation admits width $N+r$ with error $O_f(r^{-1})$. For PN networks, affine parameters or, without them, degree-one positive homogeneity recover width $N+1$; general continuous activations admit width $N+r$ with error tending to zero. The corresponding LN constructions use centered coordinate pairs, yielding widths $N+2$ and $N+2r$, respectively. We prove that $N+1$ for RMSNorm and $N+2$ for LN are also the worst-case bound for approximations, and show that applicable universal-approximation theorems for ordinary MLPs transfer to these normalized architectures. Numerical experiments recover the predicted approximation rates for both normalizations across all five settings and illustrate the shifted-ReLU finite-width obstruction.