Spatialpc reduces delay in local credit propagation over large models
SpatialPC: Deep Computation with Sublinear Local Credit Propagation
Machine Learning
Summary
When training very large computational models, it usually takes a long time for learning signals to reach the earliest parts of the system. The authors show that this delay depends more on how the parts are connected rather than just how deep the model is. They design a method called SpatialPC that arranges computations in multiple dimensions to greatly shorten this waiting time, allowing faster and more accurate learning updates. Their approach preserves accuracy on common image tasks while using fewer communication steps.
What this means in practice
- •For distributed machine learning engineers: Configure deep learning systems with multi-dimensional communication graphs to speed up credit signal propagation and reduce training latency on large models.
- •For hardware accelerator designers: Design chips and processors using SpatialPC’s graph layout to minimize local communication rounds needed for credit updates in deep networks.
Authors
Benjamin Prada, Shion Matsumoto, Ankur Arjun Mali
Abstract
On a chain of $N$ states, any local learning rule with bounded communication range needs $Θ(N)$ rounds before the loss can influence the earliest parameters, and this latency is usually attributed to depth. We show that it is a property of the communication geometry rather than of depth. For $N=m^d$ states placed at constant density in $d$ dimensions, a light-cone argument gives an $Ω(N^{1/d})$ lower bound on the number of bounded-range rounds that any local algorithm needs before its gradient estimates can be sign-correct at every parameter. We construct a unit-range, degree-$2d$ directed acyclic graph that contains a Hamiltonian computation of length $N-1$ and in which every state lies within directed distance $d(m-1)=Θ(N^{1/d})$ of the output, matching the lower bound. In the realizable-circuit model, depth and credit depth are therefore independent parameters: represented depth $Θ(N)$ and causal credit latency $Θ(N^{1/d})$ coexist, with a separation of order $N^{1-1/d}$. SpatialPC realizes this separation within predictive coding. Credit is propagated by adjoint prediction dynamics (APD), a local update whose $K$-round state on a settled acyclic graph is the truncated reverse-path sum $a^K=\sum_{k<K}(J^\top)^k b$, with error exactly $(J^\top)^K a^\star$. Graph geometry fixes when credit can first arrive at a state; the Jacobian-weighted path tail fixes how many further rounds accurate credit requires. A sparse two-dimensional Moore compiler retains the $N-1$ backbone with $\tfrac32N-2$ unit-range edges and has measured worst-node reachability $\sqrt N+2$ up to $N=65{,}536$, where a 65,535-edge backbone is reached in 258 local hops. With $T=2\sqrt N$ rounds, SpatialPC keeps finite-round task gradients aligned with settled gradients (cosine $0.93$ on MNIST and $0.97$ on Fashion-MNIST at $N=256$) and reaches $97.6\%$ and $88.1\%$ test accuracy after 20 epochs.