Mathematical analysis reveals how compositional learning arises over time

A Spectral Theory of Compositional Learning

Machine Learning

Summary

People often figure out complex ideas by combining smaller pieces they already know. This paper explains how this ability can emerge when learning from experience by analyzing simplified neural networks. The authors show how the structure of what we learn affects when and how these combinations become possible. Their work also explains why sometimes, even knowing the parts, people fail to combine them, or why one new fact can suddenly unlock many new ideas.

What this means in practice

  • For machine learning engineers: Design learning processes that identify when compositional inferences will naturally emerge from data structure.
  • For cognitive modelers: Create models that predict how and when new knowledge links abruptly unlock multiple insights in human cognition.

A theory result. No direct application yet.

Authors

Hugo Rydel

Abstract

How does compositional reasoning emerge during learning? We address this question by mathematically analyzing the learning dynamics of deep linear networks. We train these networks in structured synthetic environments and derive a theory linking the structure of experience to compositional learning. Our theory predicts when compositional inferences emerge, whether they are identifiable from the available evidence, and how new linking evidence can rapidly unlock previously unavailable inferences. These results provide a qualitative explanation for several phenomena observed in human cognition. They account for why a composition can fail despite knowing its premises, why similar compositions can emerge at different times, and how a single linking fact can suddenly enable many new inferences. Taken together, these findings establish a mathematical link between the statistical structure of experience and the development of compositional reasoning.