A Legendre polynomial based activation function: An aid for modeling of max pooling

Abstract

In this paper, we propose a novel differentiable activation function for convolutional neural networks. This function is proposed using an orthogonal Legendre polynomial based linear-in-parameter model. First, we present mathematical modeling for max pooling operation in convolutional neural networks and derive the relationship between max pooling and average pooling. The representation of max pooling using signal processing elements is presented. Further, we study approximations of various nonlinearities using the proposed linear-in-parameter model. The proposed activation function can be used as an alternative in place of existing activation functions in convolutional neural networks. Finally, the effectiveness of the proposed activation function is shown with empirical evaluations on benchmark image classification datasets using convolutional neural networks.