Faster gradients for learning from complex optimization problems
dOPT: Differentiating Conic Optimization via Geometric Reduction
Machine Learning
Summary
Many machine learning systems include decision steps that solve optimization problems with constraints, but training these systems requires calculating how changes affect outcomes, which can be very hard for complex problems called conic programs. The authors present dOPT, a new way to simplify the problem near the solution so that computing these sensitivities becomes easier and faster without relying on the original solver. Their method works well for various types of optimization problems and is faster as problems get larger, making it easier to train systems that depend on complex optimization layers.
What this means in practice
- •For machine learning engineers: Integrate fast differentiation through constrained decision layers in learning models for improved training efficiency.
- •For optimization software developers: Develop solver-agnostic tools for efficient gradient computation within conic optimization workflows.
Authors
Fengyu Yang, Connor W. Magoon, Tyler Watts, Shahar Z. Kovalsky
Abstract
Optimization layers enable the incorporation of structured constraints and decision problems into learning systems. Training such systems requires differentiating through the embedded optimization problem, which can be challenging for general conic programs. We introduce dOPT, a solver-agnostic framework that, rather than differentiating the full conic formulation, reduces it at a computed primal-dual solution to an equality-constrained quadratic program that preserves the reference solution and its first-order sensitivity. The reduction captures the local first- and second-order conic geometry relevant to differentiation and remains well defined at singular configurations. Computing solution derivatives then requires a single symmetric linear solve, independently of the forward solver. We derive explicit reductions for convex NLPs, QPs, SOCPs, and SDPs. Numerical experiments validate the computed gradients and show favorable backward-pass scalability, with substantial speedups over existing differentiable conic optimization methods as problem size increases.