Speed limit discovered for learning information in neural networks
Speed Limit for Information Acquisition in Stochastic Learning Dynamics
Machine Learning
Summary
Learning in neural networks involves changes to many adjustable parts that help the network understand data. This work looks at learning as a random process and finds a rule that limits how quickly these parts can gather new information about hidden details in the data. The authors break down this learning speed limit into parts caused by steady learning and parts caused by randomness in the training, helping explain the learning process in a new way. They check their findings on a simpler model and see that their rule accurately predicts how different hidden features appear in the network over time.
neural networksstochastic gradient descentMarkov processFisher informationlearning dynamicsinformation flowlatent variablesdriftnoiselinear regression
Authors
Shuta Kobayashi, Andreas Dechant
Abstract
Neural networks acquire internal representations through learning. In this work, we formulate stochastic gradient descent (SGD) as a Markovian stochastic process and derive a Fisher-information flow speed limit that bounds the rate at which trainable parameters can acquire information about latent variables in the data-generating process. The resulting inequality decomposes the information flow into drift and noise contributions, thereby quantifying the roles of deterministic learning forces and SGD-induced fluctuations from an information-theoretic perspective. We verify the bound in analytically tractable basis-function linear regression, where the information budget predicted by the bound reproduces the ordering and characteristic time scales with which different latent variables are encoded in the learned parameters. These results establish Fisher-information speed limits as a quantitative framework for diagnosing when and how different aspects of the data-generating mechanism are acquired during stochastic learning.