Flatness measures alone do not guarantee robust model improvements
When a Flatness Proxy Is Not a Function: Robustness Certificates and Training Interventions
Machine Learning
Summary
When making computer models robust, people often use measures of how 'flat' or smooth the model's predictions are. This paper shows that a common flatness measure does not reliably prove the model is more robust or help fix it. The researchers found that this measure can underestimate risks and be affected by certain transformations, meaning it might mislead decisions about the model's safety or how to train it. They offer corrections that better respect the model's structure but highlight limits of these flatness proxies for guaranteeing robustness.
What this means in practice
- •For machine learning engineers: Improve robustness evaluations by integrating corrected flatness measures that account for model symmetries and avoid misleading robustness claims.
- •For ai system auditors: Use refined robustness certificates to better assess risks of models after training, especially in image and algorithmic tasks.
Authors
Vicente Opazo, Jose Calatayud-Mateu, Cristobal Rojas, Cristian Buc Calderon
Abstract
A valid curvature upper bound need not justify either a robustness certificate or an intervention on an intrinsic predictor property. We demonstrate this distinction for a last-layer relative-flatness proxy used in both settings. First, empirical-risk stationarity does not eliminate pointwise first-order loss terms: at a finite global empirical-risk minimum, the retained certificate expression underestimates a loss increase by over $210\times$. We derive a globally valid, gauge-invariant feature-space repair. Second, common-row softmax shifts preserve predictions and the exact contraction while making the proxy unbounded. Even standard reference-class choices double it on average relative to the centered representation. For a single fixed-feature example with at least three classes, scalar retuning generically cannot align the induced probability updates. Row centering gives the orbit-minimized bound and restores value and full-model gradient invariance under this symmetry. Across 45 paired one-step tests on algorithmic and image models, amplified shifts separate raw-regularized predictors while quotient-regularized predictors remain aligned. Long-horizon CIFAR-10 experiments show substantial, reversible suppression of generalization, while evidence for selective delay after memorization is less consistent. Together, these results show that validity as a curvature upper bound does not by itself justify either inversion into a robustness certificate or differentiation into an intrinsic training intervention.