Sample sizes grow exponentially to estimate rotation in overcomplete ICA
Identifiable but Hard to Estimate: Rotation in Equally Spaced Overcomplete ICA
Machine LearningInformation Theory
Summary
The paper looks at how hard it is to figure out the right rotation in a certain math problem called overcomplete independent component analysis (ICA), when sources are spaced evenly. The authors found that although the rotation can be identified, the number of observations needed to estimate it accurately grows exponentially as more sources are involved. They provide mathematical ways to understand this growth and propose an estimator that works consistently. They also explore how noise affects the problem and why some features make estimation even harder.
What this means in practice
- •For signal processing engineers: Understand minimal observation requirements to accurately estimate rotations in overcomplete source separation tasks.
- •For statistical machine learning teams: Use mathematical bounds on sample sizes when designing algorithms for source separation with equally spaced components.
A theory result. No direct application yet.
Authors
Wenjun Xiao, Yinghao Fang, Hanxin Chen, Sihan Jia, Vince D Calhoun, Yuda BI
Abstract
How many observations are needed to estimate a common rotation in two-dimensional overcomplete independent component analysis? We study equally spaced mixing columns with known source count, common source law, and column norm. For a fixed non-Gaussian fourth-order Gram-Charlier source law, the rotation is identifiable modulo the column spacing. Yet the sample size for fixed relative angular accuracy grows exponentially with the source count. We characterize this sample size, up to absolute constants, by the reciprocal of the squared Hellinger distance between two rotations separated by half the column spacing. An explicit radially weighted phase estimator attains this scale uniformly over the rotation family, using the observation radius to prevent cancellation in the mean of an angular harmonic. We also determine the exact exponential decay rate of single-observation total variation for the same pair. This gives bounds on the sample size exponent; its exact value remains open. Finally, a noisy binary-source example shows how an exact zero line of the observed characteristic function can have an exponentially small normal derivative.