Hyperball optimizers match conventional optimizers in scale invariant networks
HyperTransfer: Understanding the Equivalence between Base Optimizer and Hyperball
Machine Learning
Summary
Neural network training can use different methods to update parameters, like changing both the size and direction or just the direction. The authors show that for certain networks where scaling doesn’t matter, these two methods behave the same way during training. They created a way called HyperTransfer to copy the training path of one method using only its starting settings and learning schedule, without running it. Their experiments show that this copying is very precise, and they also extended the idea to networks without the scaling property.
What this means in practice
- •For machine learning engineers: Simulate optimizer behaviors without executing them to save training time when tuning neural networks.
- •For autonomous vehicle software teams: Use equivalent Hyperball optimizers to reliably reproduce training dynamics on scale-invariant parts of vehicle perception models.
Authors
Jinghui Yuan, Hongtao Zhang, Jade Zou, Tianyu Li, Wenjie Zhou, Tianyu He, Wei Chen
Abstract
Hyperball optimizers constrain parameter norms and update only their directions, establishing a distinct paradigm for neural network optimization. Although this geometry appears fundamentally different from that of conventional Base Optimizers, which update both parameter norms and directions, we show that the two paradigms are dynamically equivalent for scale-invariant networks. Building on this equivalence, we propose HyperTransfer, which constructs a Hyperball optimizer that reproduces the dynamics of a target Base Optimizer using only its initialization and learning-rate schedule, without running the target optimizer itself. We further derive the inverse mapping and extend the framework to non-scale-invariant networks. Experiments show that both HyperTransfer and the inverse mapping produce loss trajectories nearly identical to those of their targets, suggesting that Hyperball dynamics are governed primarily by the induced effective learning-rate schedule and optimizer state.