Temporal predictive coding reveals tradeoffs in neural credit routing
Persistence, Exposure, and Credit Routing in Linear Temporal Predictive Coding
Machine LearningInformation Theory
Summary
Assigning credit to causes over time in neural networks is tricky because some signals persist but become hard to detect as errors. The authors study a linear predictive coding model and find a tradeoff between how much influence a signal keeps over time and how much error it causes for learning. They use mathematical tools to show that this tradeoff can be overcome by routing signals between persistent and exposed pathways through the network’s recurrence.
What this means in practice
- •For machine learning engineers: Design recurrent neural networks that balance signal persistence and error exposure to improve training efficiency without backpropagation.
- •For signal processing developers: Create temporal filtering algorithms that handle long delay dependencies by routing persistent and exposed modes separately for better error correction.
A theory result. No direct application yet.
Authors
Akira Tanase, Julian Gould, Jeffrey Seely
Abstract
Temporal Predictive Coding (tPC) networks offer a layer-local alternative to backpropagation. For linear tPC, we show, by applying tools from cellular sheaf theory, that temporal credit assignment is a tug-of-war between spatial displacement and temporal errors, whose ratios are modulated by persistence and exposure operators $P$ and $Q$. Spatial inference therefore gives us the effective recurrent transition $F=PA$, whilst future credit is filtered by $P^\ast$. This creates a persistence--error tradeoff in spatial modes: under non-amplifying invariant recurrence, modes that preserve influence over long delays simultaneously fail to expose that influence strongly as temporal error. Recurrence can circumvent this via routing between persistent and exposed modes.