Sigmoidal Feedforward Neural Networks Research Articles

Mutual information based weight initialization method for sigmoidal feedforward neural networks

When a sigmoidal feedforward neural network (SFNN) is trained by the gradient-based algorithms, the quality of the overall learning process strongly depends on the initial weights. To improve the algorithm stability and avoid local minima, a Mutual Information based weight initialization (MIWI) method is proposed for SFNN. The useful information contained in input variables is measured with the mutual information (MI) between input variables and output variables. The initial distribution of weights is consistent with the information distribution in the input variables. The lower and upper bounds of the weights range are calculated to ensure the neurons inputs are within the active region of sigmoid function. The MIWI method makes the initial weights close to the global optimal point with a higher probability and avoids premature saturation. The efficiency of the MIWI method is evaluated based on several benchmark problems. The experimental results show that the stability and accuracy of the proposed method are better than some other weight initialization methods.

An activation function adapting training algorithm for sigmoidal feedforward networks

The universal approximation results for sigmoidal feedforward artificial neural networks do not recommend a preferred activation function. In this paper a new activation function adapting algorithm is proposed for sigmoidal feedforward neural network training. The algorithm is compared against the backpropagation algorithm on four function approximation tasks. The results demonstrate that the proposed algorithm can be an order of magnitude faster than the backpropagation algorithm.

Evolving efficient learning algorithms for binary mappings.

Gradient descent training of sigmoidal feed-forward neural networks on binary mappings often gets stuck with some outputs totally wrong. This is because a sum-squared-error cost function leads to weight updates that depend on the derivative of the output sigmoid which goes to zero as the output approaches maximal error. Although it is easy to understand the cause, the best remedy is not so obvious. Common solutions involve modifying the training data, deviating from true gradient descent, or changing the cost function. In general, finding the best learning procedures for particular classes of problem is difficult because each usually depends on a number of interacting parameters that need to be set to optimal values for a fair comparison. In this paper I shall use simulated evolution to optimise all the relevant parameters, and come to a clear conclusion concerning the most efficient approach for learning binary mappings.

Vector space methods for sensor fusion problems

In a multiple sensor system, the sensor Sj , j=1,2, . . . ,N, outputs Y( j )?R in response to input X?[0,1], according to an unknown probability distribution PY(j )|X . The problem is to estimate a fusion function f:RN?[0,1], based on a training sample, such that the expected square error is minimized over a family of functions F that constitutes a finite-dimensional vector space. The function f* that exactly minimizes the expected error cannot be computed since the underlying distributions are unknown, and only an approximation f to f* is feasible. We estimate the sample size sufficiently to ensure that an estimator f that minimizes the empirical square error provides a close approximation to f* with a high probability. The advantages of vector space methods are twofold: (1) the sample size estimate is a simple function of the dimensionality of F and (2) the estimate f can be easily computed by the well-known least square methods in polynomial time. The results are applicable to the classical potential function method as well as to a recently proposed class of sigmoidal feedforward neural networks.