Connecting robust training methods using local curvature insights
Explaining f-Divergence-Based Regularization via Local Curvature and Sharpness-Aware Minimization
Machine LearningInformation Theory
Summary
Improving how deep learning models perform often involves making them less sensitive to small changes. The authors explain two popular techniques—divergence-based regularization and Sharpness-Aware Minimization (SAM)—by showing how both focus on flattening the landscape of possible models to avoid sharp changes. They found that these methods are closely related through the concept of curvature, which measures how quickly performance changes near a model’s parameters. Their experiments confirm that encouraging flatter solutions leads to better and more reliable predictions.
What this means in practice
- •For machine learning engineers: Design training methods that improve model robustness by incorporating curvature-based regularizers related to divergence and SAM.
- •For computer vision developers: Create more reliable image recognition models by using input-space perturbation regularization techniques that promote flat loss landscapes.
Authors
Nour Jamoussi, Marios Kountouris
Abstract
Divergence-based regularization and Sharpness-Aware Minimization (SAM) are two prominent approaches for improving generalization in deep learning, both motivated by robustness to perturbations. However, their relationship has remained largely unexplored. Building on classical second-order expansions of $f$-divergences, we show that the two methods are locally consistent under parameter-space perturbations: both induce curvature-sensitive penalties, with divergence regularization yielding a Fisher-weighted quadratic form and SAM penalizing sharpness through the dominant Hessian eigenvalue. For negative log-likelihood objectives with exponential-family output distributions, this correspondence becomes especially transparent, since the Fisher and Gauss-Newton matrices coincide. We further show that the same local geometric perspective extends to input-space perturbations, where divergence-based regularization is defined through transformations of the input. In this setting, the regularizer induces a pullback quadratic form on the input space, providing a more general perturbation framework than standard SAM while preserving the same local sensitivity interpretation. To validate the analysis empirically, we use the asymmetric $α$-skew Jensen-Shannon divergence (JSD) family as a controlled testbed. Its local curvature coefficient scales as $α(1-α)$ and is maximized at the symmetric point $α=\tfrac12$, which recovers the standard JSD. Loss-landscape visualizations in the input-perturbation regime show that stronger induced curvature penalization is associated with flatter local minima. Experiments on four benchmark datasets further demonstrate that both accuracy and negative log-likelihood are consistently best near this regime of maximal curvature penalization.