RepNet: Tackling spectral bias in deep neural networks via parameter reparameterization
2026-06-15 • Machine Learning
Machine Learning
AI summaryⓘ
The authors studied why regular deep neural networks (DNNs) struggle to learn functions that change rapidly or have many scales, a problem called spectral bias. They found that how the network starts learning—specifically, the slope and where the network splits its input—matters a lot for capturing fast oscillations. To fix this, they created RepNet, a special version of DNNs that changes the first layer's weights to better handle these issues and can adapt during training. Their tests showed that RepNet predicts tricky, wavy patterns more accurately than standard networks without much extra computing effort.
Deep Neural NetworksSpectral BiasReLU ActivationMultiscale ProblemsFunction ApproximationReparameterizationPhysics-Informed Neural Networks (PINNs)Adaptive Frequency ScalingPartial Differential EquationsOperator Learning
Authors
Yong Wang, Tao Zhou, Xuhui Meng
Abstract
Deep neural networks (DNNs) have achieved remarkable success in scientific computing, yet they often suffer from spectral bias in capturing oscillatory and multiscale behaviors. In this study, we investigate this limitation by examining the failure of shallow ReLU neural networks in fitting high-frequency functions. This observation identifies two important factors in resolving rapid oscillations: the initial slope scale and the distribution of partition points induced by the networks. Motivated by this analysis, we propose RepNet, a reparameterized DNN model for ReLU and tanh networks designed for high-frequency and multiscale problems. The key idea is to reparameterize the weights and biases in the first hidden layer, which enables effective control of the initial slope scale and provides an appropriate distribution of the initial partition points. Furthermore, treating the reparameterized weights and biases as trainable parameters allows the DNN to achieve adaptive frequency scaling during training. In addition, we derive quantitative estimates for the output and slope magnitudes of the reparameterized DNN to guide the initialization of the proposed method. Numerical experiments, including multiscale one- and four-dimensional function approximation, forward and inverse PDE problems in combination with physics-informed neural networks (PINNs), and operator learning, demonstrate that RepNet improves the predicted accuracy of vanilla DNNs in capturing highly oscillatory features with slightly additional computational cost. These results indicate that RepNet provides an effective and flexible approach for overcoming spectral bias and applying DNNs to multiscale problems.