An Energy-Efficient Online-Learning Stochastic Computational Deep Belief Network

Abstract

Deep neural networks (DNNs) are effective machine learning models to solve a large class of recognition problems, including the classification of nonlinearly separable patterns. The training of DNNs is, however, particularly difficult due to the large size and high energy consumption of the networks. Recently, stochastic computation (SC) has been considered to implement DNNs to reduce the hardware cost. However, it requires a large number of random number generators (RNGs) and long stochastic sequences that lower the energy efficiency of the network. To overcome these limitations, we propose the design of an energy-efficient deep belief network (DBN) with online learning capacity based on stochastic computation. In the SC-DBN, a reconfigurable structure is utilized to implement the fast greedy learning algorithm and an adaptive moment estimation (ADAM) circuit is designed to improve the speed of the training process. An approximate SC activation unit (A-SCAU) is further designed to implement different types of activation functions in the neurons. The A-SCAU is immune to signal correlations, so the RNGs can be shared among all neurons in the same layer with no accuracy loss. The area and energy of the proposed design are less than 5.5% and 3.7% (or 29.3% and 33.3%) of a pipelined 32-bit floating-point (or an 8-bit fixed-point) implementation. The proposed SC-DBN design achieves a higher classification accuracy compared with the fixed-point implementation. The accuracy is in a range of 0.12% to 0.37% lower than the floating-point design with a significantly lower (or slightly higher) energy consumption than the pipelined (or non-pipelined) circuit for both online learning and inference processes.