Structured features cause overfitting unlike random features in ridge regression
Structured Features Overfit Where Random Features Grok
Machine Learning
Summary
The paper explores how different kinds of features affect learning with a method called ridge regression. Xu, Vardi, and Safran previously showed that with random features, models first memorize data and then learn to generalize well. This new work finds that with more organized, structured features (like certain Fourier features), this pattern disappears and overfitting happens more easily. The authors show that increasing the complexity of features can make the model perform worse, and this relates to the geometry of the features themselves rather than just the number of features used.
What this means in practice
- •For machine learning engineers: Optimize feature selection in ridge regression to avoid overfitting caused by structured feature maps.
- •For signal processing teams: Design Fourier-based feature representations that maintain accuracy by controlling active modes to prevent performance collapse.
Tested on simulated data.
Authors
Chon-Fai Kam, Miloud Bessafi, Frederic Cadet
Abstract
Xu, Vardi and Safran (ICML 2026) prove that over-parameterized ridge regression over an unstructured random Gaussian feature map groks, with the delay between memorization and generalization growing as $1/λ$ in the weight decay. We show that on a structured feature map the same delay does not appear. For a band-limited Fourier feature map over $\mathbb{Z}_p^2$ carrying a single-character target that lies inside the expressible class, enlarging the band at fixed positive weight decay drives peak held-out accuracy monotonically from $1.00$ to $0.07$, with no memorize-then-generalize regime anywhere along the sweep. The degradation is not an interpolation effect. It sets in at capacity ratio $q/n = 0.638$, far below the interpolation threshold, on separate grounds from the exact null space that appears above it. What does have a sharp boundary is the active support. Holding the nominal dimension fixed and masking the band back to $1089$ active modes restores held-out accuracy of $1.000$ with zero variance across seeds, while the full $4225$-mode band collapses to $0.185$. The number of active modes acts through the teacher-weighted spectrum of the empirical Gram matrix and not through the capacity ratio, which makes this a statement about feature geometry and not a restatement of double descent.