Random synaptic feedback weights support error backpropagation for deep learning.

Timothy P Lillicrap,Daniel Cownden,Colin J Akerman,Douglas B Tweed

doi:10.1038/ncomms13276

Abstract

The brain processes information through multiple layers of neurons. This deep architecture is representationally powerful, but complicates learning because it is difficult to identify the responsible neurons when a mistake is made. In machine learning, the backpropagation algorithm assigns blame by multiplying error signals with all the synaptic weights on each neuron's axon and further downstream. However, this involves a precise, symmetric backward connectivity pattern, which is thought to be impossible in the brain. Here we demonstrate that this strong architectural constraint is not required for effective error propagation. We present a surprisingly simple mechanism that assigns blame by multiplying errors by even random synaptic weights. This mechanism can transmit teaching signals across multiple layers of neurons and performs as effectively as backpropagation on a variety of tasks. Our results help reopen questions about how the brain could use error signals and dispel long-held assumptions about algorithmic constraints on learning.

Highlights

The brain processes information through multiple layers of neurons
Backprop multiplies error signals e by the weight matrix WT, which is the transpose of the forward synaptic connections, W (Fig.1b)
Backprop computes feedback by multiplying error signals e by the weight matrix WT, which is the transpose of the forward synaptic connections W

Summary

Introduction

The brain processes information through multiple layers of neurons. This deep architecture is representationally powerful, but complicates learning because it is difficult to identify the responsible neurons when a mistake is made. The backpropagation algorithm assigns blame by multiplying error signals with all the synaptic weights on each neuron’s axon and further downstream This involves a precise, symmetric backward connectivity pattern, which is thought to be impossible in the brain. As an alternative to sending error information antidromically, it have been suggested that errors could instead be fed back through a second network[4,21,23,24,25,29,30,31,32] Most of these approaches either assume that forward and feedback connections are symmetric, or they propose more intricate learning rules for the backward weights that maintain precise symmetry. Whilst the brain does exhibit widespread reciprocal connectivity that would be consistent with the transfer of error information across layers, it is not believed to exhibit such precise patterns of reciprocal connectivity[21]

Objectives

Methods

Results

Conclusion