Neuroevolutionary based convolutional neural network with adaptive activation functions

Abstract

Deep convolutional neural networks are one of the most successful types of neural networks widely used in image processing and pattern recognition. These networks involve many tunable parameters that influence network performance drastically. Among them, the proposed method of this paper focuses on the role of activation function in these networks, while the idea of adaptive activation functions is further developed by utilizing the neuroevolutionary technique. Considering several basic function to be combined in a non-linear manner, the proposed method attempts to construct an adaptive function by the help of Genetics Algorithm (GA) technique, while the selected basic functions by GA and the learned combination coefficients are adapted to the input data. As the network optimizer and the learning rate parameter are tightly related to the network activation functions, they are also included in the GA evolutionary process to be selected such that they are highly in coherence with the selected basic functions. Experiments done on the classification of CT brain images and the MNIST hand written digits dataset clearly confirm the efficiency of the proposed idea and the role of proper adaptive activation functions in extending the capabilities of neural networks.