Sharpness aware mode connectivity improves neural network robustness
SAM-on-the-Curve: Sharpness-Aware Mode Connectivity for Robust Weight-Space Interpolation
Machine Learning
Summary
Deep learning models trained separately but performing similarly can be connected by smooth curves in their settings, called mode connectivity. The authors found that simply having a low-loss curve does not guarantee the model will perform well when the input data changes in unexpected ways. They propose a new method, Sharp Mode Connectivity (SMC), that ensures the entire curve neighborhood has stable performance by avoiding sharp peaks in the loss landscape. This method leads to better accuracy under difficult test conditions and even finds points on the curve that outperform the original models.
What this means in practice
- •For machine learning engineers: Improve reliability of deep learning models under noisy or corrupted data by using sharpness-aware interpolation between trained model weights.
- •For computer vision developers: Enhance image classification robustness by applying sharpness-aware mode connectivity to models dealing with real-world visual corruptions like blur.
Authors
Alejandro Calatrava, Xu Zhang, Ren Wang
Abstract
Deep neural networks that are independently trained to similar performance can be connected by low-loss parametric curves in weight space, a phenomenon known as Mode Connectivity (MC). This geometric property underpins practical techniques such as weight averaging, model ensembling, and model merging. We argue that low-loss connectivity is an incomplete geometric criterion: it controls loss only along a one-dimensional trajectory while leaving the surrounding weight-space neighborhood unconstrained, so the optimized curve may traverse sharp ridges that become fragile under distribution shift. We therefore reformulate mode connectivity as a neighborhood-robust path optimization problem, seeking a curve whose entire local neighborhood maintains low loss. We propose Sharp Mode Connectivity (SMC), which applies a first-order sharpness-aware approximation to the resulting minimax functional, enforcing flatness along the entire curve rather than only on it. We derive a practical optimization algorithm for connectivity paths under this sharpness-aware objective. Under severe blur corruptions from CIFAR-10-C, SMC achieves up to 6.09\% absolute accuracy improvement over standard MC. Remarkably, SMC produces negative loss barriers, meaning that models obtained at interior points of the optimized path can outperform the average endpoint loss. These results, validated across ResNet-18, VGG16-BN, and ViT-Tiny on CIFAR-10 and ImageNet-100, establish path-wise flatness as a practical principle for robust weight-space interpolation.