New regression method improves prediction under rare data shifts
CVaR anchor regression protects against rare shifts
Machine Learning
Summary
Machine learning models often struggle when they see rare but big changes in data after training. The authors study a new regression approach called CVaR anchor regression that focuses on protecting predictions against these rare changes without losing accuracy in usual cases. This method looks at the worst-case average error in the tails of data shifts rather than the average over all data, which helps it handle rare shifts better. They prove guarantees for its performance in certain linear models and show it works well on real taxi data.
What this means in practice
- •For data science teams: Improve prediction reliability when rare but large shifts appear in training data, reducing failures in unusual environments.
- •For transportation analysts: Enhance forecasting models for transportation data with rare but impactful shifts, such as demand spikes or anomalies in taxi services.
Tested on one dataset.
Authors
Malte Londschien
Abstract
We study prediction in new environments when training data contain rare, large shifts. Anchor regression penalizes the average of the squared mean residual across environments. It protects against shifts in an ellipsoid determined by the second moment of the training shifts. Covering rare shifts may therefore require a large penalty, expanding the ellipsoid in every direction and reducing accuracy on common environments. We propose CVaR anchor regression, which replaces the average of the squared mean residuals with a tail average. Unlike CVaR or GroupDRO applied directly to prediction risks, it does not give environments more weight solely because their noise levels are high. We prove an exact worst-case risk guarantee under a linear structural model that allows for heteroscedastic noise. For discrete environments, decreasing the CVaR tail fraction expands the robustness set from an ellipsoid to a scaled convex hull of the training shifts and their negatives. A separate parameter controls its scale. Examples show how the method can improve protection against rare shifts while retaining accuracy on common environments. We illustrate the method on New York City taxi data.