Abstract
Deep Neural Networks (DNNs) have become the tool of choice for machine learning practitioners today. One important aspect of designing a neural network is the choice of the activation function to be used at the neurons of the different layers. In this work, we introduce a four-output activation function called the Reflected Rectified Linear Unit (RReLU) activation which considers both a feature and its negation during computation. Our activation function is “sparse”, in that only two of the four possible outputs are active at a given time. We test our activation function on the standard MNIST and CIFAR-10 datasets, which are classification problems, as well as on a novel Computational Fluid Dynamics (CFD) dataset which is posed as a regression problem. On the baseline network for the MNIST dataset, having two hidden layers, our activation function improves the validation accuracy from 0.09 to 0.97 compared to the well-known ReLU activation. For the CIFAR-10 dataset, we use a deep baseline network that achieves 0.78 validation accuracy with 20 epochs but overfits the data. Using the RReLU activation, we can achieve the same accuracy without overfitting the data. For the CFD dataset, we show that the RReLU activation can reduce the number of epochs from 100 (using ReLU) to 10 while obtaining the same levels of performance.
Highlights
Background and Related WorkActivation functions have been studied for a very long time, since the early days of neural networks
This paper is organized as follows: In Section 2, we present a detailed study of the various activation functions that have been studied in the context of neural networks; in Section 3, we formally introduce the fouroutput variation of the Rectified Linear Unit (ReLU) activation and study some of its properties
If the single neuron does nothing but passes on this input to its output, the output of the neuron is wTx, the operation that is being performed at the neuron is nothing but a simple aggregation and if this happens at each of the nodes of the hidden layer, the neural network becomes a system that outputs a linear combination of its inputs, which is not much of a learning
Summary
Background and Related WorkActivation functions have been studied for a very long time, since the early days of neural networks. Let us suppose that the input to a shallow (single hidden layer) neural network is given by the vector x. If the single neuron does nothing but passes on this input to its output, the output of the neuron is wTx, the operation that is being performed at the neuron is nothing but a simple aggregation and if this happens at each of the nodes of the hidden layer, the neural network becomes a system that outputs a linear combination of its inputs, which is not much of a learning. Note that if is the identity mapping, the output to a node is the linear combination of the input values. This is known as a linear activation and as we have seen above, it does not do much
Sign-in/Register to view highlights & summary Talk to us
Join us for a 30 min session where you can share your feedback and ask us any queries you have
Schedule a callDisclaimer: All third-party content on this website/platform is and will remain the property of their respective owners and is provided on "as is" basis without any warranties, express or implied. Use of third-party content does not indicate any affiliation, sponsorship with or endorsement by them. Any references to third-party content is to identify the corresponding services and shall be considered fair use under The CopyrightLaw.
