Complex step size adjustment improves training of complex neural networks
AURA: Angular Update Rate Adaptation for training complex-valued neural networks
Machine Learning
Summary
Training neural networks that use complex numbers can be tricky because regular training methods don’t fully consider how complex numbers behave. The authors created a way called AURA that adjusts how much each network parameter changes during training by watching patterns in previous updates. This method works with existing training techniques, making learning faster and more stable without needing extra guesses. Tests on different problems showed AURA often helps neural networks learn better with little added work.
What this means in practice
- •For machine learning engineers: Enhance training efficiency of complex-valued neural networks by integrating adaptive step sizes that consider complex number properties.
- •For signal processing developers: Use improved optimization for neural networks handling complex signals to achieve faster and more reliable training.
Authors
Enrico Ballini, Allan Peter Engsig-Karup, Tito Andriollo
Abstract
Complex-valued neural networks (CVNNs) are increasingly adopted for complex-valued data; however, they are often trained with first-order optimizers inherited from the real-valued case. The efficiency of these methods depends largely on the step size, and their step-size rules ignore the angular information available in the complex plane. We address step-size adaptation in the complex domain by introducing AURA (Angular Update Rate Adaptation), a per-parameter step-size adaptation that can be added on top of any first-order optimizer, and removed from it, without altering its update direction. AURA measures the agreement between consecutive updates of each complex parameter, in length, alignment, and sense of rotation, and enlarges the step when they are consistent and reduces it when they are not. It requires no additional gradient evaluations and only inexpensive vector operations per step. We combine AURA with Adam and Muon and compare the resulting methods with well-known first-order optimizers on four test cases of increasing complexity, ranging from the approximation of scalar complex functions to physics-informed training. Fully connected neural networks are used throughout this work. All hyperparameters other than the step size are held fixed across test cases; for one case, we also tune the hyperparameters of each optimizer under the same budget. Our empirical tests show that AURA improves the convergence of its base optimizer in most cases with a small per-step overhead, and we identify the conditions under which it fails to do so.