Backpropagation trains deep networks by sending a single error signal backward through every layer in a locked sequence of forward pass, backward pass, and weight update. Biological brains show no mechanism for this global phase‑locking, so researchers look for locally‑updatable alternatives. Predictive coding (PC) treats each layer’s activity as an optimization variable and updates weights with a Hebbian‑like rule, but the supervisory signal must diffuse through many layers. In deep, narrow nets the signal fades before it reaches the input, causing PC to lag far behind backprop.
Sakana AI’s Augmented Lagrangian Predictive Coding (PC‑ALM) adds a layer‑wise Lagrange multiplier to the PC energy. During inference the network alternates a primal step on activations and a dual step that accumulates the local prediction error. This dual step acts as an integral term, turning the pure proportional PC update into a PI controller per layer. As a result the error propagates as a wave with constant speed instead of diffusing like heat, so credit reaches the earliest layers even when the network is very deep.
The method remains strictly local: each layer needs only its own activation, prediction, and multiplier. A JAX reference implementation (MIT licensed) runs on CPU and reproduces the paper’s width‑depth sweep. Tests on MNIST and Fashion‑MNIST show that a residual MLP with 1000 layers stays within about two percentage points of backprop accuracy, while vanilla PC drops sharply. Gradient alignment improves from 0.60 to 0.91 in the reference cell, and the approach also helps PC on vision benchmarks such as ResNet‑18 on CIFAR‑10 and Tiny ImageNet.
PC‑ALM therefore offers a practical, biologically plausible training rule that closes the gap to backprop without requiring a global backward pass. It can be swapped into existing code as a drop‑in replacement for the weight‑update step.
#AI #MachineLearning #DeepLearning #Neuroscience #PredictiveCoding #BackpropAlternative