Is the reconstruction loss culprit? An attempt to outperform JEPA
Machine Learning
Summary
The authors compare two types of models that learn to represent data: JEPA-style predictive models and autoencoders that try to reconstruct data. They find that while JEPA seems better at handling noise at first, autoencoder problems are actually tied to how they pick features and handle complexity. To fix this, the authors create a new gated autoencoder that picks out the most predictable parts of the data. This new model works well across different noise levels and does as well or better than JEPA in their tests.
Authors
Alexey Potapov, Oleg Shcherbakov, Ivan Kravchenko
Abstract
We evaluate JEPA-style predictive representation learning versus reconstruction-based autoencoders on a controlled "TV-series" linear dynamical system with known latent state and a single noise parameter. While an initial comparison suggests JEPA is markedly more robust to noise, further diagnostics show that autoencoder failures are strongly influenced by asymmetries in objectives and by bottleneck/component-selection effects (confirmed by PCA baselines). Motivated by these findings, we introduce gated predictive autoencoders that learn to select predictable components, mimicking the beneficial feature-selection behavior observed in over-parameterized PCA. On this toy testbed, the proposed gated model is stable across noise levels and matches or outperforms JEPA.