Geometry measurements fail to reliably choose best predictive model
A Generalisation Signal Need Not Be a Model-Selection Signal
Machine LearningArtificial Intelligence
Summary
Choosing the best trained model based on usual testing might not work well when the real-world use is different from training conditions. The authors tested if looking inside the model’s mathematical shape (geometry) can help pick the best one instead. They found that some geometry-based signals do relate to how well a model generalizes, but these signals do not consistently identify the best model for new conditions. This shows that just because a measure signals generalization ability does not mean it helps pick the best model.
What this means in practice
- •For computational biology teams: Help choose more reliable criteria when selecting models for biological prediction tasks outside trained conditions.
- •For machine learning engineers: Avoid relying solely on geometric curvature measures when evaluating models for deployment shifts to improve model selection robustness.
Authors
Aditya Nagarsekar, M P Ashish Bhat, Aadi Nesarkar, Vrishti Godhwani, Rahul Yedida, Aditya Challa, Danda Sravan, Snehanshu Saha
Abstract
Model selection in computational biology often relies on validation data drawn from the training regime, even when deployment lies outside it. When validation no longer preserves which model is best, a natural alternative is to rank candidates using properties of the trained network itself. We test this idea using a novel, forward-only proxy motivated by the norm of the Hessian, alongside common Hessian measures, across molecular property, protein fitness, and drug-response tasks. Contrary to our hypothesis, geometry does not become more useful as validation Spearman correlation deteriorates: augmenting validation helps some shifts but significantly harms others. More surprisingly, the proxy still correlates with generalisation gap on most tasks even when Hessian trace and top-eigenvalue relationships are weak or reversed, yet this signal does not reliably identify the deployment-best model. A curvature bound need not preserve cross-model rankings, and low geometric scores can even favour collapsed predictors. Thus, a generalisation signal need not be a model-selection signal.