Neural networks learn simple waves before complex patterns naturally
Why Shallow Neural Networks Learn Sinusoidal Components First: An Orthogonal Polynomial Dynamics Framework
Machine LearningArtificial Intelligence
Summary
Neural networks often learn simple, smooth patterns like slow waves before more complex fast ones. The paper explains why this happens by looking at how the network changes during training using math involving special waves and polynomials. It also shows how the type of network parts, like activation functions, affects this learning order. This work helps explain a common quirk of neural networks with broad conditions and more precise math.
What this means in practice
- •For machine learning engineers: Design networks that better match learning speeds of different signal components by understanding spectral bias origins and activation function roles.
- •For signal processing developers: Improve algorithms for signal reconstruction by integrating insights on why networks prioritize sinusoidal components during training.
A theory result. No direct application yet.
Authors
Yanyang Xiao
Abstract
Neural networks often learn low-frequency components before high-frequency ones --- a phenomenon called spectral bias or termed F-principle. Many numerical studies have demonstrated this phenomenon, and theoretical studies have sought to explain it. However, several goals remain insufficiently addressed: (1) explaining spectral bias asymptotically, including its origin and why the preferred components are generally sinusoidal; (2) extending the explanation to general activation functions, such as sigmoid and ReLU; (3) making weak assumptions about the data distribution; (4) avoiding aliasing effects, such as those arising in finite-domain Fourier analysis; and (5) developing a theoretical framework applicable to arbitrary networks. We address these goals theoretically by analyzing the eigenvalue decay and eigenvector shapes of the dynamics matrix H, which arises when we expand the network output as a general Fourier series (GFS) in an orthogonal polynomial basis and express the gradient-descent dynamics of the GFS coefficients as ordinary differential equations. The graded structure of H enables an intuitive explanation of spectral bias, while orthogonal polynomial theory helps us analyze how learning bias depends on the activation function and, potentially, the network architecture.