Online supervised learning reveals trade-offs in dimensionality reduction
Online Supervised Dimension Reduction with Random Features: Diagnostics and Computational Trade-offs
Machine Learning
Summary
Reducing data to fewer dimensions helps computers learn faster and clearer patterns, but getting the best simplified view doesn’t always mean better predictions. The authors studied a method combining random features with an online updating rule to understand how well it finds important data directions and how accurate the predictions are. They found that optimizing the math objective alone doesn’t guarantee better practical results, and quicker calculations may come with errors in the learned directions. Their work shows there is a balance between speed, accuracy, and understanding data geometry that can’t be judged just by final scores.
What this means in practice
- •For machine learning engineers: Choose between online updates or exact computations to manage trade-offs in speed and accuracy when reducing data dimensions for predictive models.
- •For data scientists: Diagnose whether improved training objective scores actually correspond to better predictive features or just better optimization in high-dimensional supervised learning tasks.
Authors
Zhenlin Yao, Wei Xiong
Abstract
Accurate optimization of a supervised spectral objective need not produce an accurate population subspace or a better predictive representation. We investigate these distinctions for Online Kernel Supervised Principal Component Analysis (OKSPCA), which combines a centered cross-moment in finite random-feature coordinates with an Adam-style orthonormal basis update for an established objective. Fixed-map consistency, concentration and perturbation results describe the estimator and its exact subspace; same-target comparisons then assess the practical iterate separately. Across six predictive benchmarks, performance depends on the declared pipeline: replacing the tracker with the exact empirical target leaves the two regression deficits largely unchanged. Direct classification-rank models capture nearly all terminal objective energy on average, but a saved intermediate state exhibits substantial geometric deviation; a controlled sample-size study further separates empirical accuracy from population recovery. In distinct numerical-service workloads, exact on-request computation is faster in the tested classification settings, whereas Adam saves time relative to the tested full thin-SVD service for some dense wider-regression requests, alongside persistent geometric error. These diagnostics limit explanations based solely on terminal optimization accuracy and distinguish numerical cost from quality, rank coverage and freshness; they establish neither practical-tracker convergence nor predictive or deployment benefits from basis availability.