Deep learning reveals complex geometric structures in data spaces
The Geometry of Manifold Learning: Singular foliations via Deep Learning models
Machine Learning
Summary
Neural networks learn from data, but it's unclear exactly how they organize the information they receive. The authors study a mathematical way to describe the shapes or patterns that a trained neural network creates in the space of its inputs. They focus on a tricky case caused by certain functions inside the network that are not smooth, which makes the math harder. By smoothing the problem just enough, they show these patterns form a special kind of geometric structure called a singular foliation, providing new insight into how neural nets sort data.
What this means in practice
- •For machine learning engineers: Use the geometric insights to design neural network architectures that better capture data structure despite non-smooth activations.
- •For data visualization specialists: Develop tools that map high-dimensional neural network inputs into geometric patterns reflecting learned data organization.
A theory result. No direct application yet.
Authors
Gabriele Novelli, Eliot Tron, Emanuele Latini, Rita Fioresi
Abstract
Deep learning algorithms have become widely used and successful across a wide range of tasks. A key challenge is determining whether these algorithms truly capture the intrinsic geometry of their training data in order to better understand what they learn from data. In this theoretical work, we explore a new geometric structure defined on the data space by a trained neural network through the Data Information Matrix (DIM), a variation of the Fisher Information Matrix. This new geometric structure is a foliation and the main focus of this work is to explore the non-smooth case of ReLU activation functions, where traditional geometric tools often fail. The foliation is defined through a distribution spanned by the columns of the DIM. We first prove that this distribution is non-smooth in the sense of distributions, rendering the existence of an associated foliation much more complex. To circumvent this issue, we introduce a smoothed version of this distribution that differs from the original only on a set of measure zero. Finally, we prove that this smoothed distribution induces a singular foliation on the data space, which is based on the data the network has been trained with. Our theoretical contributions provide a geometric framework to better understand how neural networks organize their input space in non-smooth settings, bridging differential geometry and deep learning theory.