AdamX optimizes machine learning training using cosine similarity
AdamX: Cosine similarity meets gradient descent
Machine Learning
Summary
Training machine learning models requires adjusting parameters step-by-step, which can be tricky and slow. The paper introduces AdamX, a new method that uses the angle (cosine similarity) between updates to better control how much each step changes the model. This helps make training smoother, especially in the early phases, and works well across different datasets and models. The authors show that AdamX competes well with existing popular methods by requiring fewer training steps to reach good results.
What this means in practice
- •For machine learning engineers: Improve training efficiency by integrating AdamX to reduce epochs needed for model convergence across diverse architectures.
- •For ai product developers: Use AdamX to speed up model updates in applications requiring rapid retraining or fine-tuning with limited compute.
Authors
Francisco Caldas, Ruben Belo, Cláudia Soares
Abstract
We introduce AdamX, a first-order optimizer that incorporates cosine similarity as an adaptive mechanism for controlling update magnitudes. The proposed method is scalable, model-agnostic, and straightforward to integrate into existing training pipelines. We further introduce a variance rectification scheme that promotes smoother optimization during the early stages of training. Overall, we provide empirical evidence that AdamX achieves competitive convergence rates across a range of benchmark datasets and architectures. Performance is evaluated in terms of the number of epochs required to reach predefined performance thresholds under a fixed hyperparameter budget. Code and Experiments available at: https://github.com/FranciscoCaldas/adamX.