Learning conditional expectations without fixed bases using functional Newton updates
Learning Conditional Expectation Operators via Functional Newton Updates
Machine Learning
Summary
Predicting how one variable depends on another can be tricky when you want to find the main patterns efficiently. The authors came up with a new method called FSNM that learns these patterns without relying on preset building blocks or fixed frameworks. Their approach repeatedly improves its guesses through updates that solve easier problems, eventually finding a simple but accurate representation. Tests show this method can recover key structures and answer many related questions with the same learned model.
What this means in practice
- •For machine learning engineers: Build models that estimate conditional relationships without choosing fixed bases or kernels beforehand, improving flexibility and reuse across queries.
- •For statistical data analysts: Extract low-rank approximations of joint probability relationships to better understand underlying data dependencies in complex datasets.
Authors
Thiago Ramos, Alek Fröhlich, Daniel Perazzo, Massimiliano Pontil
Abstract
We introduce the Functional Spectral-Newton Method (FSNM) for learning the leading singular structure of a conditional expectation operator without fixing a basis or reproducing kernel Hilbert space. FSNM fits a low-rank representation of the centered joint-to-product density ratio kernel by alternating functional Newton updates. Each update reduces to a preconditioned regression, which we approximate with vector-valued regression trees in a stagewise boosting procedure. At the population level, we establish descent and an $O(1/T)$ best-iterate block-stationarity rate under a relative weak-learner accuracy condition, and show that every nondegenerate local minimum over the full centered $L^2$ spaces is a globally optimal rank-$d$ approximation. Synthetic experiments show that FSNM recovers a low-rank density ratio and its leading spectral structure, and that the same learned kernel can answer multiple conditional queries without refitting.