The effect of amplitude modification in S-shaped activation functions on neural network regression

Abstract

Time series forecasting using multilayer feed-forward neural networks (FNN) is potential to give high accuracy. Several factors influence the accuracy. One of them is the choice of activation functions (AFs). There are several AFs commonly used in FNN with their specific characteristics, such as bounded type AFs. They include sigmoid, softsign, arctan, and tanh. This paper investigates the effect of the amplitude in the bounded AFs on the FNNs’ accuracy. The theoretical investigations use simplified FNN models: linear equation and linear combination. The results show that the higher amplitudes give higher accuracy than typical amplitudes in softsign, arctan, and tanh AFs. However, in sigmoid AF, the amplitude changes do not influence the accuracy. These theoretical results are supported by experiments using the FNN model for time series prediction of 10 foreign exchanges from different continents compared to the US dollar. Based on the experiments, the optimum amplitude of the AFs should be high, that is greater or equal to 100 times of the maximum input values to the FNN, and the accuracy gains up to 3–10 times.