AdamX improves training speed using cosine similarity and gradient descent
AdamX: Cosine similarity meets gradient descent
Machine Learning
Summary
Training computer models often requires adjusting parameters efficiently to learn from data. Francisco Caldas and colleagues created AdamX, a new method that changes how these adjustments are made by measuring angles between updates, helping control step sizes more adaptively. They also added a feature to reduce randomness in early training, which makes learning smoother. Tests show AdamX performs well across different models and datasets without extra tuning effort.
Adam optimizergradient descentcosine similarityoptimization algorithmsvariance rectificationtraining convergencemachine learningparameter updatesbenchmark datasets
Authors
Francisco Caldas, Ruben Belo, Cláudia Soares
Abstract
We introduce AdamX, a first-order optimizer that incorporates cosine similarity as an adaptive mechanism for controlling update magnitudes. The proposed method is scalable, model-agnostic, and straightforward to integrate into existing training pipelines. We further introduce a variance rectification scheme that promotes smoother optimization during the early stages of training. Overall, we provide empirical evidence that AdamX achieves competitive convergence rates across a range of benchmark datasets and architectures. Performance is evaluated in terms of the number of epochs required to reach predefined performance thresholds under a fixed hyperparameter budget. Code and Experiments available at: https://github.com/FranciscoCaldas/adamX.