Linear recursive feature machines reliably learn features from noisy data
Exact Dynamics and Finite-Sample Trajectory Recovery of Linear Recursive Feature Machines
Machine Learning
Summary
Learning good representations from noisy data is tricky, but this paper studies a simple model called linear recursive feature machines that repeatedly updates features to improve predictions. The authors analyze how these features change during training and show they get closer to the optimal solution as more data is used. This analysis extends to situations with noise and multiple outputs, showing the error decreases predictably as sample size grows. They also test their findings on real text and gene expression data to illustrate these learned features.
What this means in practice
- •For machine learning engineers: Improve training stability and feature quality in multi-output regression tasks under noise using linear recursive feature machines with guaranteed finite-sample convergence.
- •For bioinformatics teams: Extract meaningful features from noisy single-cell gene-expression data for downstream analysis with a theoretically grounded linear model.
Authors
Andrew Cheng, Bobak T. Kiani, Yue M. Lu, Adityanarayanan Radhakrishnan
Abstract
Recursive feature machines (RFMs) learn representations of data by alternating between fitting a predictor to a dataset and updating features of that predictor using the average gradient outer product (AGOP). Connections between AGOPs and feature learning in neural networks motivate linear RFMs as a simple setting for analyzing how representations evolve during training. Here, we study the dynamics and statistics of linear RFM in noisy multi-output regression with isotropic sub-Gaussian input data and targets generated by a low-rank teacher matrix of dimension $d$. We extend the known connection between linear RFM and iteratively reweighted least squares from the interpolating setting to ridge-regularized multi-output regression with noise. We show that the learned feature matrix remains close to its infinite-data ideal counterpart at every iteration. Namely, for $n$ samples, we show the error in the feature matrix decays as $O(\sqrt{d/n})$ with high probability. Experiments on real-world text and single-cell gene-expression data illustrate the features learned by this simple linear model.