Backpropagation works well for deep learning but it requires a backward pass that uses the exact transpose of the forward weight matrix. This weight transport problem has no clear biological counterpart because real neurons cannot copy and send weights in reverse. Sakana AI’s paper addresses this by proposing a learning rule that obeys Dale’s principle—neurons are either purely excitatory or purely inhibitory—and avoids any need for transposed weights or random feedback matrices.
The core idea is Error Diffusion, a local update rule that uses only three signals: presynaptic activity, the derivative of the postsynaptic activation, and a single global error sign. To make this rule work for multi‑class tasks the authors split every layer into two streams, an excitatory stream (p) and an inhibitory stream (n). All learnable weight matrices are kept non‑negative; the inhibitory influence is built into the network structure through fixed minus signs between the streams. This dual‑stream design respects Dale’s principle while preserving the expressive power needed for complex data.
To route the error signal to the correct hidden units they introduce modulo error routing: each hidden unit i receives the error from output channel r(i) = i mod C, where C is the number of classes. This fixed, structured mapping replaces the random feedback used in earlier algorithms and eliminates weight transport entirely.
Three practical innovations boost performance on standard benchmarks. First, layer‑specific sigmoid widths keep activation derivatives large enough to prevent vanishing signals. Second, batch‑centered class error makes the error zero‑mean per class, counteracting the imbalance between correct and incorrect labels. Third, an asymmetric initialization sets excitatory weights larger than inhibitory ones, establishing a favorable excitation/inhibition balance.
With these modifications Error Diffusion reaches 96.7 % on MNIST and 61.7 % on CIFAR‑10, the first time the rule has trained convolutional networks. In reinforcement learning, combining Error Diffusion with Proximal Policy Optimization (ED‑PPO) matches or exceeds baseline methods on locomotion tasks while staying fully Dale‑compliant.
The approach shows that biologically plausible learning can compete with standard backpropagation, offering a path toward hardware‑friendly, non‑negative implementations in neuromorphic and photonic systems.
#AI #MachineLearning #DeepLearning #Neuroscience #RL #DalePrinciple