EvE optimizer ranks neural net setups faster than Adam
EvE: An Alternate Optimizer to Adam
Machine LearningNeural and Evolutionary Computing
Summary
Training neural networks usually takes a lot of time to figure out if a setup works well, which slows down trying many different designs. The authors introduce EvE, a new method that quickly tests and ranks network setups by combining a population-based search with short efforts of classic training only when needed. EvE manages to find good configurations faster than the widely used Adam optimizer, especially when time or computational resources are limited. However, EvE may not replace Adam for final training because it trades some quality for speed.
What this means in practice
- •For machine learning engineers: Rank neural network hyperparameter and architecture configurations faster under tight evaluation budgets.
- •For data science platform teams: Speed up automated model tuning by integrating EvE as a proxy optimizer to prune configurations early.
Authors
Shashank Raj, Kalyanmoy Deb
Abstract
Adam and its variants dominate neural network training, but a single run only reveals whether a configuration works well after most of its budget is spent, a poor fit for hyperparameter or architecture search, where configurations must be ranked cheaply and pruned early. We introduce EvE (Evolutionary Explorer), a steady-state, population-of-four differential evolution (DE) optimizer with a targeted Adam fallback: each iteration proposes one candidate via DE, running a short burst of gradient descent only if the DE step fails to improve on the incumbent. Selection is greedy, so on a deterministic objective the best-so-far value is provably monotone non-increasing, and since gradients are used only as a targeted rescue, per-iteration cost stays within a constant factor of a single Adam step regardless of dimension. Under a fixed, evaluation-cost-matched budget, EvE wins or ties Adam on 76% of 70 (problem, dimension) cells across seven scalable benchmarks up to one million variables. On three real neural-network tasks (an MLP on MNIST, and LoRA fine-tuning of a 1.5B-parameter language model on two datasets) EvE finishes the same charged budget 1.7-3.9x faster, at a modest cost in final quality (about one accuracy point on MNIST, 9-11% higher relative test loss on the two fine-tuning tasks; on GSM8K, Adam is about 5 accuracy points more accurate, and fine-tuning lowers accuracy below the base model for both). Inside successive halving on UCI Adult, EvE completes hyperparameter and architecture searches 3.1-3.5x faster, ranking configurations about as consistently with Adam as Adam does with itself across seeds (Kendall's tau 0.66-0.69). EvE is not a total replacement for Adam as a final-stage trainer, but a fast, gradient-aware proxy for the search-heavy, budget-constrained regime one level up.