Real analytic nonlinear ICA recovers hidden factors for kinked sources
Kinks vs. Smoothness: Identifiability of Real Analytic nICA for Laplace-like Sources
Machine Learning
Summary
Many machine learning methods try to find the hidden factors behind complex data like images. The challenge is to recover the true original factors rather than a scrambled version. The authors show that when the hidden factors have certain sharp changes (kinks), like in Laplace-distributed data, and the model is based on smooth, real analytic functions, the true factors can be exactly identified. They confirm this with experiments using common tools like Normalizing Flows and Variational Autoencoders. This means some existing systems can reliably find meaningful hidden factors when the data fits these conditions.
What this means in practice
- •For machine learning engineers: Design latent factor recovery models that guarantee exact factor identification when source factors exhibit Laplace-like kinks.
- •For computer vision teams: Improve interpretability of image attributes by recovering stable latent factors using real analytic based nonlinear ICA.
Tested on simulated data.
Authors
Isaac Manring, Kejun Huang
Abstract
Many machine learning systems try to explain complex data - like images or financial time series - in terms of hidden, independent factors that generated them. Recovering the true underlying factors, rather than some scrambled version of them, is the central challenge of nonlinear Independent Component Analysis (nICA). We prove identifiability (exact recovery) up to trivial ambiguities for real analytic generating functions when source probability density functions have a finite number of discontinuities in the first derivative. The Laplace distribution is the most prominent example satisfying this assumption. Our proof relies on the contrast between kinks in the source distribution and the smoothness of real analytic functions. Real analytic functions comprise a broad class of generating mechanisms, and can be approximated with Normalizing Flows or Variational Autoencoders with standard activation functions (e.g., tanh, softplus, GELU), so our result applies with minimal changes to existing training pipelines. We perform experiments on real and synthetic data with both Normalizing Flows and Variational Auto-Encoders demonstrating their identifiability properties. In experiments on CelebA data we recover several interpretable latent factors controlling unique attributes across the dataset.