Performance Analysis of Fully Connected Parametric Activation Functions

Abstract

Modern studies have shown that the choice of activation function can significantly affect the performance of several learning methods and deep networks. The activation function plays an important role in solving the nonlinear problem, and various nonlinear fully connected activation functions have been studied. In this paper, we propose a combined parametric activation function that can improve the performance of a fully connected Artificial intelligence methods and neural network. Combined parametric activation functions can be created by simply adding parametric activation functions. The parametric activation function is a function that can be optimized in the direction of minimizing the loss function by applying a appropriate parameter that converts the scale and location of the activation function according to the input data. The development of Artificial Neural Networks (ANNs) has achieved a lot of fruitful results so far, and we know that activation function is one of the principal factors which will affect the performance of the networks. In this paper, we discussed about the impact of varying model width and depth on robustness, the impact of using learnable parametric activation functions (PAFs).