An Adaptive Offset Activation Function for CNN Image Classification Tasks

Abstract

The performance of the activation function in convolutional neural networks is directly related to the model’s image classification accuracy. The rectified linear unit (ReLU) activation function has been extensively used in image classification models but has significant shortcomings, including low classification accuracy. The performance of a series of parametric activation functions has made parameter addition a popular research avenue for improving the performance of activation functions in recent years, and excellent progress has been achieved. Existing parametric activation functions often focus on assigning a different slope to the negative part of the activation function and still involve the negative value alone in the activation function calculation, without considering the impact of linking the negative value to the positive value on the performance of the activation function. As a result, this work proposes a novel parametric right-shift activation function, the adaptive offset activation function (AOAF). By inserting an adaptive parameter (the mean value of the input feature tensor) and two custom ReLU parameters, the negative parameters previously driven to zero by ReLU can be turned into positive parameters with lower weight and participate in CNN feature extraction. We compared the performance of the suggested activation function to the performance of a selection of typical activation functions using four distinct public datasets. Compared with ReLU, the average classification accuracy of our proposed activation function improved by 3.82%, 0.6%, 1.02%, and 4.8% for the four datasets, respectively.