Robust sparse regression method balances stability and simplicity in models
Sparse Regression Distilled from a Single Robust Fit
Machine Learning
Summary
Sometimes when analyzing data, noisy or corrupted responses make it hard to find simple, reliable explanations. The authors developed a new method that starts with a strong but complex fit and carefully simplifies it into a sparse and stable model without losing accuracy. This approach helps keep predictions consistent even when the data shifts slightly, though the simplest models may sacrifice some detail. They tested their method on simulated and real data, showing it balances robustness, sparsity, and prediction well.
What this means in practice
- •For data science teams: Build predictive models that remain stable despite noisy or shifted training data, balancing model simplicity and accuracy.
- •For financial analysts: Develop sparse regression models for high-dimensional economic data to retain reliable predictions under data contamination or outliers.
Authors
Wooyoung Shin, Seunghwan Park
Abstract
Robust linear fits can resist response contamination yet remain too dense or unstable for useful global explanations. We propose penalized distillation, which fits a smoothly clipped absolute deviation (SCAD) estimator to a robust initial estimator's empirical fitted surface along a safeguarded coordinate-descent path and evaluates candidate states separately for fidelity, parsimony, perturbation stability, and held-out prediction. The new results attach to the states the algorithm actually computes. Conditional on a fixed uncontaminated design, deterministic bounds transfer response-replacement boundedness from the initial fit to every retained path state. Turning to fixed dimension, we characterize the oracle-support branch by its empirical-Gram projection and influence function, give conditions for covariance-weighted least-squares approximation equivalence, and establish a path-conditional generalized information criterion. By contrast, at large dimension-to-sample ratios the full-coordinate robust fit collapses without warning, and screening restores the construction. Under a sure-screening framework, the robustness bound and the support and selection guarantees transfer to the screened fit. Simulations separate robustness transfer from support recovery, efficiency, and computation across the dimension-to-sample ratio, with p up to 240, and the signal density, which isolates what the sparse stage adds once the screen over-selects. In a duplicate-grouped superconductivity study, the distilled estimator remains predictively stable under prespecified training-response shifts but retains 66.8--68.8 of 81 slopes. Stronger sparsification reduces the model to 12.6--14.0 slopes only at visible fidelity and prediction cost. Distillation therefore preserves predictive stability on these data without substantiating a compact coordinate-level explanation.