Linearized sigmoidal activation: A novel activation function with tractable non-linear characteristics to boost representation capability

Abstract

Activation functions play a crucial role in discriminative capabilities of the deep neural networks. They are also one of the main reasons for revival of the neural networks. Although, recent activation functions provide solution to vanishing and exploding gradient problems, but they do not have sufficient capacity to model non-linear data. This paper proposes a novel activation function, which imparts neural networks with capability to model non-linear dependencies in data. Behavior of proposed activation function remains non-saturating even with non-linear structure. The activation function provides distinct activation behaviors to different data range segments. The paper presents two different variants of proposed function. The first, “linear sigmoidal activation” function is a fixed structure activation function with function coefficients defined at the start of model design. Whereas second, “adaptive linear sigmoidal activation” function is a trainable function which can adapt itself according to the complexity of the given data. Both of the proposed models are tested against the state of the art activation functions on benchmark datasets (CIFAR-10, MNIST, SVHN, FER-2013). The proposed activation function is able to outperforms every known activation functions in all the tests.