Maximal coding rate reduction has fundamental limits in out-of-distribution generalisation
On the Limits of Maximal Coding Rate Reduction for Out-of-Distribution Generalisation
Machine Learning
Summary
Deep learning tries to create models that work well even when situations change, but some methods can still fail badly. The authors studied a method called maximal coding rate reduction (MCR²) that aims to learn clear and structured ways to represent different categories. They found that MCR² can completely fail to predict correctly when the environment changes, even if stable features exist. Adding other well-known techniques does not fix this problem, showing that more assumptions or new ideas are needed to make MCR² reliable for changing conditions.
What this means in practice
- •For machine learning engineers: Recognize fundamental limits of maximal coding rate reduction when deploying models in changing environments to avoid unexpected failures.
- •For autonomous vehicle teams: Avoid relying solely on maximal coding rate reduction-based methods for perception modules needing robust predictions across diverse driving scenarios.
A theory result. No direct application yet.
Authors
Menghui Zhou, Gaoshan Bi, Vitaveska Lanfranchi, Po Yang
Abstract
Substantial efforts have been devoted to making deep learning objectives, representations, and architectures interpretable, with the goal of improving the safety, robustness, and generalisation of learning systems in diverse real-world applications. The recently proposed maximal coding rate reduction ($\mathrm{MCR}^{2}$) offers a promising information-theoretic framework for learning structured, discriminative representations of class-wise submanifolds and has inspired interpretable white-box architectures. However, we observe that $\mathrm{MCR}^{2}$ can completely fail under distribution shift, motivating our study of its out-of-distribution (OOD) generalisation limits. We establish two limitations of $\mathrm{MCR}^{2}$ for OOD generalisation. First, the $\mathrm{MCR}^{2}$ objective alone can admit complete prediction failure: a representation based entirely on unstable environmental features can achieve the global coding optimum yet fail completely after correlation reversal, despite an available perfectly stable feature. This exact-optimum example includes test inputs that cannot occur during training. Even when every possible test input can also occur during training, coding quality can be arbitrarily close to optimal while prediction error is arbitrarily close to 100%. Second, directly incorporating the invariance principle underlying widely successful invariant risk minimisation (IRM) and risk extrapolation (REx) does not eliminate this failure. The failing representation admits the same optimal coding operator across training environments, showing that shared coding optimality does not ensure stable prediction. Reliable OOD guarantees for $\mathrm{MCR}^{2}$ therefore require additional new assumptions or learning principles that establish stable predictive relationships across environments.