Deep networks learn efficiently with 4 bit predictive coding
Error-Based Bidirectional Predictive Coding Enables Deep, 4-Bit Predictive Coding Networks
Machine Learning
Summary
Training very deep predictive coding networks can be very slow and inaccurate because the signals weaken as they pass through layers, especially when using low precision numbers. The authors show that by combining two recent techniques—using errors rather than states for inference, and bidirectional predictive coding—they can keep signals strong and stable even in deep networks with very low (4-bit) precision. Their approach lets the network learn as well as full precision models and produce high-quality outputs while using less computational resources. This could help build hardware that runs such networks faster and more efficiently.
What this means in practice
- •For hardware accelerator designers: Develop efficient hardware chips optimized for deep predictive coding networks running with 4-bit precision while maintaining accuracy.$Commercial implications: Enables design of low-power, compact AI accelerators for inference and training in edge devices where resource use and energy are limited.
- •For embedded system developers: Implement neural networks with low-precision predictive coding to perform fast and flexible inference on limited hardware resources like IoT devices.
Authors
Garrett Kinman, Zeljko Zilic
Abstract
Predictive coding networks (PCNs) promise properties not offered by backpropagation-trained networks: learning from purely local information, a single network that both discriminates and generates, and flexible inference by energy minimization. However, their iterative inference is expensive, making hardware acceleration desirable, and a useful, efficient accelerator needs both depth and aggressive quantization. Canonical state-based PC delivers neither: its inference signal decays exponentially with depth, so deep models fail to train, and low precision makes it worse. We show that pairing two recent methods carries the full PCN property set into the deep, 4-bit regime: bidirectional PC (bPC), which unifies discrimination and generation, and the error-based reparameterization (ePC), which optimizes prediction errors rather than states during inference. Error optimization delivers the output signal to every layer in a single backward pass, removing the signal decay; one additional element, global-norm clipping of the error gradients, stabilizes the error dynamics at depth. The resulting ePC+bPC model trains stably up to at least eight layers and retains its full-precision accuracy and clean class-conditional generation under W4A4E4 quantization-aware training, across FP4 and three posit formats. On six-layer networks, where state-based bPC collapses to chance, ePC+bPC classifies more than 50 accuracy points above it after one epoch of training, with class-conditional generations 5-9x better in RMSE. 4-bit quantization is nearly free: full-precision and 4-bit accuracies differ by at most ~5 points, and by under two in the best-performing formats. We characterize the two failure modes the method overcomes, bounded signal decay in state-based PC and error divergence in unregularized error optimization, and discuss implications for a 4-bit-native PCN accelerator.