Mode dynamics reveal how neural network learning changes outputs

Mode Dynamics: A Framework for Neural Network Learning

Machine Learning

Summary

Neural networks learn by adjusting many parameters, but understanding how these changes affect what the model outputs is complex. The authors propose Mode Dynamics, which breaks down parameter updates into collective patterns (modes) and links each to how it changes the model's behavior. They show that a few key modes explain most output changes and that these modes depend on the task and data. This framework helps clarify how learning progresses inside neural networks.

What this means in practice

Authors

Hossein Honarvar

Abstract

We introduce Mode Dynamics, a framework that connects coordinated parameter motion during neural network learning to changes in model outputs. Starting from a recorded parameter trajectory, we decompose its updates into collective modes and map each mode to its state-dependent output effect. The framework organizes this link through three mechanisms. Decomposition identifies collective modes in parameter motion and their time-varying amplitudes. Reconstruction tests whether the effects induced by those modes recover local and accumulated output change. Concentration quantifies how behavioral contribution is distributed across modes. Controlled scalar regression and next-token experiments directly test these mechanisms. They show that small sets of leading modes reconstruct behavior, induced effects depend on the training path, and modal complexity varies with task and data. We then evaluate the framework on held-out image and language-model outputs using CIFAR-100 autoencoding and a public Pythia-70M pretraining trajectory. Across these settings, the results connect compact modal subspaces of recorded parameter motion to evolving behavior.