Abstract
While the backpropagation of error algorithm enables deep neural network training, it implies (i) bidirectional synaptic weight transport and (ii) update locking until the forward and backward passes are completed. Not only do these constraints preclude biological plausibility, but they also hinder the development of low-cost adaptive smart sensors at the edge, as they severely constrain memory accesses and entail buffering overhead. In this work, we show that the one-hot-encoded labels provided in supervised classification problems, denoted as targets, can be viewed as a proxy for the error sign. Therefore, their fixed random projections enable a layerwise feedforward training of the hidden layers, thus solving the weight transport and update locking problems while relaxing the computational and memory requirements. Based on these observations, we propose the direct random target projection (DRTP) algorithm and demonstrate that it provides a tradeoff between accuracy and computational cost that is suitable for adaptive edge computing devices.
Highlights
Artificial neural networks (ANNs) were proposed as a first step toward bio-inspired computation by emulating the way the brain processes information with densely-interconnected neurons and synapses as computational and memory elements, respectively (Rosenblatt, 1962; Bassett and Bullmore, 2006)
We use an error-sign-based version of Direct feedback alignment (DFA), subsequently denoted as sign-based direct feedback alignment (sDFA), in which the error vector is replaced by the error sign in the global feedback pathway
DRTP Is a Simplified Version of Error-Sign-Based DFA As we have shown that sDFA solves both the weight transport and the update locking problems in classification tasks, we propose the direct random target projection (DRTP) algorithm, illustrated in Figure 1D and written in pseudocode in Algorithm 1, as a simplified version of sDFA that enhances both performance and Algorithm 1: Pseudocode for the direct random target projection (DRTP) algorithm. k ∈ [1, K], k ∈ N, denotes the layer index and Wk, bk, Bk and fk(·) denote the trainable forward weights and biases, the fixed random connectivity matrices and the activation function of the k-th hidden layer, respectively
Summary
Artificial neural networks (ANNs) were proposed as a first step toward bio-inspired computation by emulating the way the brain processes information with densely-interconnected neurons and synapses as computational and memory elements, respectively (Rosenblatt, 1962; Bassett and Bullmore, 2006). In order to train ANNs, it is necessary to identify how much each neuron contributed to the output error, a problem referred to as the credit assignment (Minsky, 1961). The backpropagation of error (BP) algorithm (Rumelhart et al, 1986) allowed solving the credit assignment problem for multi-layer ANNs, enabling the development of deep networks for applications ranging from computer vision (Krizhevsky et al, 2012; LeCun et al, 2015; He et al, 2016) to natural language processing (Hinton et al, 2012; Amodei et al, 2016). Beyond implying a perfect and instantaneous communication of parameters between the feedforward and feedback pathways, error backpropagation requires each layer to have full knowledge of all the weights in the downstream layers, making BP a non-local algorithm for both weight and error information.
Sign-in/Register to view highlights & summary Sign-in/Register to access full text options 
Talk to us
Join us for a 30 min session where you can share your feedback and ask us any queries you have
Schedule a callDisclaimer: All third-party content on this website/platform is and will remain the property of their respective owners and is provided on "as is" basis without any warranties, express or implied. Use of third-party content does not indicate any affiliation, sponsorship with or endorsement by them. Any references to third-party content is to identify the corresponding services and shall be considered fair use under The CopyrightLaw.
