Emulation of Spline Networks Through Approximation of Polynomials and Step Function of Neural Networks with Cosine Modulated Symmetric Exponential Function

Abstract

This study proves that the neural network, using the cosine modulated symmetric exponential function which is a non-monotonic function, can emulate the spline networks by approximating the polynomials and step functions. This means that the network with cosine modulated symmetric exponential function is equivalent to the spline networks. It is also equivalent to the neural network, which uses sigmoidal, hyperbolic tangent, and Gaussian activation function, as proved by DasGupta and Schnitger. In the multi-network structure, the cosine modulated symmetric exponential function has the capability to make more local hills than any other functions. In the neural network that uses this function, it has the capability to quickly localize the input space pattern, even though it makes a fewer number of layers. On the other hand, the monotonic function needs a greater number of layers to make these local hills. Therefore, in the training for the pattern classification of the neural network, we need a greater number of units and epochs. This is connected to the training speed of the neural network for the pattern classification, which also indicates the capabilities of the network. For the capacity test of the pattern classification in the cosine modulated symmetric exponential function, we have used the Cascade-Correlation neural network. Cascade-Correlation is a supervised learning algorithm that automatically determines the size and topology of the network. The Cascade-Correlation adds new hidden units one by one and creates a multi-layer structure in which each unit is in a hidden layer. In this experiment, the two benchmark problems have been used: one is the iris plant classification problem; the other is the tic-tac-toe endgame problem. The results are compared with those obtained with other activation functions. In this experiment, the evaluation items, such as the number of epochs, produced hidden units, listing of the run time, and the average crossings per second of ten trials on the training set of the problem, have been compared. For instance, In the experiment of the iris plants classification problem, the CosExp function has recorded about 53 % of the average epochs number when compared to the sigmoid function, and for the number of the hidden units, it is approximately 54 %. In the tic-tac-toe problem experiment, the average number of epochs and hidden units produced during the process has been reduced approximately by one thirds more with the CosExp function than with other activation functions. Accordingly, learning has been improved three times faster. The results of the experiments show that performance can be improved very significantly by using the cosine modulated symmetric exponential function as the activation function in neural networks with a predetermined set of parameters.