Generalized Convexity and Smoothness via Conjugate Duality: Optimization Theory for Deep Neural Networks
2026-08-10 • Machine Learning
Machine Learning
AI summaryⓘ
The authors address why deep neural network training works well even though traditional optimization theories don’t fully explain it. They create a new framework that generalizes key concepts like convexity and smoothness using advanced math tools, allowing them to describe both simple and complex training problems together. They propose new versions of gradient descent that guarantee certain convergence rates and show how network design and training choices affect learning performance. Their experiments with various networks and datasets support their theoretical findings.
deep neural networkstochastic gradient descentconvexitysmoothnessLegendre functionsconvex conjugationgradient descentcomposite optimizationJacobianconvergence rate
Authors
Binchuan Qi
Abstract
Deep neural network (DNN) training with stochastic gradient descent (SGD) and its variants achieves strong empirical performance, yet classical optimization theory does not fully explain this success. This limitation arises because conventional analyses rely on assumptions such as differentiability, convexity, or smoothness, which are often violated by DNN objectives. In this paper, we establish a unified optimization framework for DNN training by generalizing classical convexity and smoothness through Legendre functions and convex conjugation. Specifically, we introduce $\mathcal{H}(ψ)$-convexity and $\mathcal{H}(Ψ)$-smoothness, which unify convex and non-convex as well as smooth and non-smooth objectives within a single formalism and reveal a natural duality between generalized smoothness and convexity. Building on these generalized properties, we introduce generalized gradient descent (GD) and generalized SGD through convex conjugation. We theoretically prove that generalized GD admits an optimal learning rate of exactly $1$, and derive rigorous gradient-energy-based convergence rates for both proposed optimizers. We further reformulate DNN training as a composite optimization problem, demonstrating that its convergence relies on jointly reducing the gradient energy and controlling the induced norm of the network Jacobian. To characterize the practical influences of network architectures and training configurations, we introduce the gradient correlation factor and model capacity risk, and quantitatively analyze how architectural designs, batch size, and model capacity shape training convergence. Extensive experiments across diverse network architectures, datasets, optimizers, and loss functions validate our theoretical bounds and demonstrate precise alignment between our theoretical predictions and empirical training dynamics.