De-floored Principal Component Regression: When Rank Selection Alone Is Insufficient for Prediction
Machine Learning
Summary
The authors study a method called principal component regression (PCR), which simplifies complex data by focusing on important parts but can overestimate some key values. They introduce a new approach, de-floored PCR (dPCR), that adjusts these overestimates by subtracting a value called the "floor." Their results show that dPCR performs better in prediction tasks, especially when this floor can be accurately estimated and removed. They also provide mathematical explanations and conditions under which their method works well compared to traditional PCR.
Authors
Peng Zhao
Abstract
Principal component regression (PCR) regularizes high-dimensional prediction by choosing a spectral cutoff, but rank selection cannot correct systematic inflation of the retained empirical eigenvalues. We study clean Gaussian random designs in which the aggregate covariance tail creates a nearly scalar sample-space floor comparable to the predictive head scale. De-floored principal component regression (dPCR) retains the cutoff and subtracts an estimated floor from the retained denominators. We prove an ordinary-PCR prediction-risk lower bound uniform over all ranks and a high-probability dPCR upper bound. When the floor is sharp and inexpensive to remove in population prediction risk, the conditional risk of dPCR is asymptotically negligible relative to that of the best ordinary PCR rank. An exact risk decomposition explains the separation: denominator inflation is governed by first spectral mass, whereas the clean prediction cost of correction is governed by squared spectral mass. A same-sample trimmed-mean floor estimate attains the oracle dPCR upper-bound rate at a prespecified rank, and the separation persists under approximate predictive alignment when the tail prediction-energy fraction vanishes. Separate pointwise fixed-aspect formulas show that the risk-optimal positive scalar correction improves rank-$1$ PCR, whereas mean-floor subtraction is generally not optimal for a broad Marchenko--Pastur bulk.