Improving neural network optimization by lifting nonconvex problems
Taming Nonconvexity with Epigraph Lifting
Machine Learning
Summary
When using neural networks for solving tricky problems, the usual methods can get stuck in bad solutions because the math is complicated. The authors propose a new way to rewrite the problem that breaks it into simpler parts, so it’s easier to find the best answer. They also introduce a step-by-step method that gradually improves the solution by building on earlier steps. Tested on different tasks like image cleanup, their approach consistently finds better solutions than standard techniques starting from the same point.
What this means in practice
- •For computer vision engineers: Improve image reconstruction tasks such as denoising and inpainting by optimizing neural network models to avoid poor local minima.
- •For machine learning practitioners: Enable more reliable training of neural-network-based regularizers through a structured penalty method that recovers original objectives exactly.
Authors
Clara Lage, Hok Shing Wong, Matthias Ehrhardt
Abstract
We consider inverse problems regularized by a trained neural network with \emph{convex activations} and \emph{unrestricted weights}. The resulting optimization problem is generally nonconvex, and gradient-based methods may converge to poor local minima. We introduce a \emph{penalized epigraph lifting} of the network and show that the lifted problem admits a \emph{difference-of-convex (DC) decomposition}. For Lipschitz continuous activations, we establish \emph{finite exactness}: the original objective is recovered exactly at a finite and explicitly computable penalty threshold $\barρ$. Building on this structure, we propose \emph{H-DC} (Homotopic DC), a homotopic scheme that solves a sequence of DC subproblems with increasing $ρ$, warm-starting each level from the previous solution. Numerical experiments on synthetic problems, a manifold denoiser regularizer, and image inpainting with a learned Mumford--Shah regularizer show that H-DC consistently escapes poor local minima and reaches solutions with lower objective values than gradient-based methods from the same initialization.