Flatness alone does not drive better generalization in neural nets

Do Flatter Minima Drive Better Generalization? An Algorithmic Separation in Grokking

Machine Learning

Summary

Knowing whether a computer model that is good at remembering data will also be good at learning general patterns is hard. The authors study how ‘flatness’ of the training error impacts this. They find that just making a model's training landscape flatter does not guarantee it will learn generalizable rules. However, when combined with techniques like weight decay, flatter solutions can speed up learning of generalizable patterns. Their math model shows how flatness and weight decay interact, clarifying when and why flatness helps.

What this means in practice

  • For machine learning engineers: Improve training schedules for neural networks by combining flatness-inducing methods with weight decay to reach generalizing solutions faster.
  • For ai model trainers: Diagnose when flatness metrics can and cannot predict model generalization, helping avoid false optimism in model tuning.

Authors

Mohnish Harwani

Abstract

Flat loss landscapes have long been linked to better generalization in neural networks. However, its role as a causal mechanism for generalization is less established. Grokking provides an unique testbed to understand this distinction: models are prone to fit observed data using non-generalizing structure and remain in that regime for prolonged periods, transitioning to generalization only under particular training conditions. In this work, we study whether flat loss landscapes can act as a driving mechanism in this transition. While recent work has argued for flatness as a necessary geometric condition for this transition, we find that biasing training toward flatter solutions using sharpness-aware minimization (SAM) is insufficient to reliably induce this transition, despite producing flatter solutions. However, when SAM is paired with mechanisms that drive generalization such as weight decay, an interesting property emerges: SAM can accelerate the transition to generalizing solutions by up to 4x at the epoch-level. We theoretically untangle this relationship between SAM and weight decay using a minimal interpolating two-layer ReLU model with both memorizing and generalizing solutions. We show that even in this simple setup, flatness alone cannot distinguish a memorizing solution from a generalizing one, while weight decay favors generalizing solutions. However, under a local stability analysis, there exists a window where a memorizing interpolant is locally stable under gradient descent but unstable under SAM in the low-norm regime, which can explain SAM's ability to accelerate this transition. Overall, our results provide a more interpretable account of the role of flatness in driving generalization, especially in settings where models are vulnerable to minimizing loss through learning non-generalizing structure.