Logish: A new nonlinear nonmonotonic activation function for convolutional neural network

Abstract

Activation function is an important component of the convolutional neural network. Recently, nonlinear nonmonotonic activation functions such as Swish and Mish have illustrated good performance in deep learning structures. In this paper, we propose a new nonlinear nonmonotonic activation function called Logish, which can be represented by f(x)=x·ln[1+sigmoid(x)]. Firstly, we take the logarithmic operation to reduce the numerical range of sigmoid(x)+1, then we employ variable x to make the negative output have a strong regularization effect. Furthermore, we evaluate the image classification performance of Logish and its variant f(x)=αx·ln[1+sigmoid(βx)] in simple and complex networks with top–1 accuracy. Experimental results demonstrate that Logish’s variant (α=1,β=10) can achieve 94.8% top-1 accuracy with ResNet–50 network on CIFAR 10 dataset, and can reach 99.24% top-1 accuracy with DenseNet on MNIST dataset and 88.52% top-1 accuracy with SE-Inception-v4 network on SVHN dataset respectively. It is higher than the Sigmoid, Tanh, ReLU, Swish and Mish activation functions in the corresponding dataset. It also verifies the performance and effectiveness of Logish.