A novel scaled-gamma-tanh (SGT) activation function in 3D CNN applied for MRI classification

Abstract

Listen

Translate article

Activation functions in the neural network are responsible for ‘firing’ the nodes in it. In a deep neural network they ‘activate’ the features to reduce feature redundancy and learn the complex pattern by adding non-linearity in the network to learn task-specific goals. In this paper, we propose a simple and interesting activation function based on the combination of scaled gamma correction and hyperbolic tangent function, which we call Scaled Gamma Tanh (SGT) activation. The proposed activation function is applied in two steps, first is the calculation of gamma version as y = f(x) = axα for x < 0 and y = f(x) = bxβ for x ≥ 0, second is obtaining the squashed value as z = tanh(y). The variables a and b are user-defined constant values whereas alpha and beta are channel-based learnable parameters. We analyzed the behavior of the proposed SGT activation function against other popular activation functions like ReLU, Leaky-ReLU, and tanh along with their role to confront vanishing/exploding gradient problems. For this, we implemented the SGT activation functions in a 3D Convolutional neural network (CNN) for the classification of magnetic resonance imaging (MRIs). More importantly to support our proposed idea we have presented a thorough analysis via histogram of inputs and outputs in activation layers along with weights/bias plot and t-SNE (t-Distributed Stochastic Neighbor Embedding) projection of fully connected layer for the trained CNN models. Our results in MRI classification show SGT outperforms standard ReLU and tanh activation in all cases i.e., final validation accuracy, final validation loss, test accuracy, Cohen’s kappa score, and Precision.