CMOS Implementations of Rectified Linear Activation Function

Abstract

Deep Neural Networks have become an increasingly favourite choice for a variety of machine learning tasks. Two important components are largely responsible for this success, improved neural network functionalities, and availability of suitable hardware for training large complex networks. Using these types of novel networks and functions, Deep Neural Networks have been shown to be very highly efficient for various classification tasks. As the next level of optimization, dedicated ASIC and FPGA ICs are being developed, to realize Deep Neural Networks. This provides an additional level of performance optimization beyond traditional software-based implementations. Towards this direction, in this work, we have developed CMOS circuits for realizing the highly popular Rectified Linear (ReLu) activation function. The ReLu activation function has largely replaced the traditional sigmoid activation function due to better learning rates and reduced computational requirements. With dedicated CMOS implementations of such functions, we get better operating speed with lower power consumption, leading to improved real-time implementations of classification tasks.