The underlying mechanisms of alignment in error backpropagation through arbitrary weights

Abstract

Understanding the mechanisms by which plasticity in millions of synapses in the brain is orchestrated to achieve behavioral and cognitive goals is a fundamental question in neuroscience. In this regard, insights from learning methods in artificial neural networks (ANNs) and in particular supervised learning using backpropagation (BP) seem inspiring. However, the implementation of BP requires exact matching between forward and backward weights, which is unrealistic given the known connectivity pattern in the brain (known as the “weight transport problem”). Notably, it has been shown that under certain conditions, error BackPropagation Through Arbitrary Weights (BP-TAW) can lead to a partial alignment between forward and backward weights (weight alignment or WA). This learning algorithm, which is also known as feedback alignment (FA), can result in surprisingly good degrees of accuracy in simple classification tasks. However, the underlying mechanisms and mathematical basis of WA are not thoroughly understood. In this work, we demonstrate the mathematical basis of WA and answer the question of why and in what conditions WA occurs. We show that the occurrence of WA in ANNs is induced by statistical properties of the output and error signals of neurons, such as autocorrelation and cross-correlation, and can happen even in the absence of learning or reduction of the loss function. Moreover, we show that WA can be improved significantly by limiting the norm of input weights to neurons and that such a weight normalization (WN) method can improve the classification accuracy of BP-TAW. The findings presented can be used to further improve the performance of BP-TAW and open new ways for exploring possible learning mechanisms in biological neural networks without exact matching between forward and backward weights.