AdaCubic: An Adaptive Cubic Regularization Optimizer for Deep Learning
2026-04-10 • Machine Learning
Machine Learning
AI summaryⓘ
The authors introduce AdaCubic, a new method that smartly adjusts how much it uses a cubic term in an optimization process to find better solutions more efficiently. They use a clever technique called Hutchinson's method to estimate complex information while saving computing power. Their tests show AdaCubic works well across different fields and doesn’t need users to tweak settings much, unlike some other methods. This makes it a useful tool for people working on training deep learning models. AdaCubic is also the first to apply this kind of cubic regularization in large-scale deep learning.
cubic regularizationNewton's methodHessian matrixHutchinson's methodoptimization algorithmdeep learninglocal convergencehyperparameter tuningadaptive optimizationscalability
Authors
Ioannis Tsingalis, Constantine Kotropoulos, Corentin Briat
Abstract
A novel regularization technique, AdaCubic, is proposed that adapts the weight of the cubic term. The heart of AdaCubic is an auxiliary optimization problem with cubic constraints that dynamically adjusts the weight of the cubic term in Newton's cubic regularized method. We use Hutchinson's method to approximate the Hessian matrix, thereby reducing computational cost. We demonstrate that AdaCubic inherits the cubically regularized Newton method's local convergence guarantees. Our experiments in Computer Vision, Natural Language Processing, and Signal Processing tasks demonstrate that AdaCubic outperforms or competes with several widely used optimizers. Unlike other adaptive algorithms that require hyperparameter fine-tuning, AdaCubic is evaluated with a fixed set of hyperparameters, rendering it a highly attractive optimizer in settings where fine-tuning is infeasible. This makes AdaCubic an attractive option for researchers and practitioners alike. To our knowledge, AdaCubic is the first optimizer to leverage cubic regularization in scalable deep learning applications.