Benign overfitting depends on joint input interaction not separate inputs
Benign Overfitting under Heterogeneous Input Fusion
Machine Learning
Summary
Sometimes, machine learning models can fit noisy data without losing accuracy, a situation called benign overfitting. This paper studies what happens when data comes from different types of inputs combined together, instead of just one. The authors find that whether overfitting is harmless depends on how the combined inputs interact, not just on their individual traits. Also, the impact of combining inputs can differ for regression and classification tasks. Therefore, understanding the relationship between input blocks is crucial for predicting overfitting outcomes.
What this means in practice
- •For machine learning engineers: Predict when combining different data sources in linear models leads to harmless overfitting for better model reliability.
- •For data integration specialists: Assess joint data characteristics to avoid harmful effects when merging features from multiple sources in predictive models.
A theory result. No direct application yet.
Authors
Houzhen Liu, Xiaobo Xia
Abstract
Benign overfitting is extensively studied when learning from a single high-dimensional input, but its behavior under heterogeneous input fusion remains largely unexplored. We study this question for minimum-norm linear interpolation under a heterogeneous Gaussian design, comparing two statistically dependent input blocks with their fusion while holding the underlying population task fixed. For regression, we identify a full-spectrum covariance certificate whose asymptotic status is independent of the cutoff threshold and prove that it is preserved by every positive-semidefinite joint covariance consistent with the two marginals. This protection is sharp, yet it does not extend to all benign regression problems: outside the certified regime, two benign marginals can have a harmful fusion. For one-sparse Gaussian classification, benignity in the regular regime is characterized by the balance between surviving predictive signal and nuisance contamination. Fusion can move these two quantities in opposite directions, and within this model class every marginal-to-joint benign/non-benign pattern is attainable. We further show that the same fused input can have qualitatively different effects on regression and classification. These results establish that benign overfitting under heterogeneous fusion is determined by the joint signal and spectral geometry created by input interaction, rather than by marginal benignity alone.