Hyperball optimizers work like base optimizers for scale-invariant networks
HyperTransfer: Understanding the Equivalence between Base Optimizer and Hyperball
Machine Learning
Summary
Optimizing neural networks usually means adjusting both how big certain parameters are and which way they point. Hyperball optimizers, however, only change the direction, keeping size fixed. The authors found that for a kind of network where scale doesn't matter, these two approaches end up doing the same thing. They created a method called HyperTransfer that can mimic one optimizer using the other without rerunning the original process. This helps us understand that controlling learning rates and optimizer state are key to how these optimizers behave.
neural networksoptimizerHyperballbase optimizerscale-invariantparameter normlearning rateoptimizer dynamicsloss trajectoryHyperTransfer
Authors
Jinghui Yuan, Hongtao Zhang, Jade Zou, Tianyu Li, Wenjie Zhou, Tianyu He, Wei Chen
Abstract
Hyperball optimizers constrain parameter norms and update only their directions, establishing a distinct paradigm for neural network optimization. Although this geometry appears fundamentally different from that of conventional Base Optimizers, which update both parameter norms and directions, we show that the two paradigms are dynamically equivalent for scale-invariant networks. Building on this equivalence, we propose HyperTransfer, which constructs a Hyperball optimizer that reproduces the dynamics of a target Base Optimizer using only its initialization and learning-rate schedule, without running the target optimizer itself. We further derive the inverse mapping and extend the framework to non-scale-invariant networks. Experiments show that both HyperTransfer and the inverse mapping produce loss trajectories nearly identical to those of their targets, suggesting that Hyperball dynamics are governed primarily by the induced effective learning-rate schedule and optimizer state.